lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge remote-tracking branch 'origin/working' into working

Browse source
This commit is contained in:
lei.chen 2026-08-01 03:02:49 +00:00
commit 93ebea572f
98 changed files with 110312 additions and 1176 deletions
Download patch file Download diff file Expand all files Collapse all files

11
src/controller/CLAUDE.md
View file

@ -62,10 +62,16 @@ The file `raft/raft.tla` contains Diego Ongaro's TLA+ specification of the Raft
| `RequestVote(i,j)` | `raft_on_vote_request` / reply path | `raft.c:106,169` |
| `BecomeLeader(i)` | `raft_cb_become_leader` | `ha.c:347` |
| `AdvanceCommitIndex(i)` | `repl_wait_majority_ack` → advance `committed_lsn` | `repl.c:338` |
| `AppendEntries(i,j)` | `OPCode_nop` heartbeat + WAL streaming | `ha.c:2880`, `config_repl.c:1489` |
| `AppendEntries(i,j)` — production send (LIVE) | `repl_broadcast` (per-entry fan-out on append, NACK-driven retry via `ha_raft_on_repl_nack`) + `ha_stream_wal_to_standby` (catch-up-range streaming) — the sole production `AppendEntries`-equivalent send/retry/backoff mechanism (raft-adapter-conformance Req 3.3) | `repl.c:295`, `config_repl.c:2351` |
| `AppendEntries(i,j)` — nextIndex-driven sender (DORMANT, documented) | `raft_send_append_entries` is fully protocol-conformant but has ZERO production callers (only `raft_sim.c` and an `XCE_TESTING` test helper); its registered `on_send_append_entries` callback `ha_cb_send_append_entries_stub` is reachable only through it. Kept as an intentionally-dormant reference implementation (retire/wire-in both rejected, see `research.md`), guarded against silent regression by `ha_send_append_entries_stub_call_count` staying 0 while the production path fires (`test_ha_core_safety.c` task 4.2) | `src/lib/raft/raft.c:595`, `ha.c:2590` (stub), `ha.c:2588` (guard counter) |
| `AppendEntries(i,j)` reply — WAL catch-up reject | Every WAL catch-up rejection sends an explicit `WalAckReply{success=0}` via `ha_send_wal_ack_reply_ex` back to the primary — never a silent drop (bug-1479; reply-construction boundary regression-locked by raft-adapter-conformance task 5.1) | `ha.c:3155` (fn), `ha.c:3190` (accept call site), `ha.c:3326` (reject call site) |
| `AppendEntries(i,j)` reply — configuration-push accept/reject | Every `handle_repl_config_push` outcome — all 4 reject branches plus the success/`-EAGAIN`-activation-deferred tail — sends exactly one explicit `ReplConfigPushReply{accepted, term}` via `send_config_push_reply`, so the primary can distinguish an active reject from a lost push (raft-adapter-conformance Req 2.2-2.5); the reply's `term` field feeds the primary's `raft_on_higher_term_observed` term-adoption alongside the heartbeat and WAL-NACK vectors (Req 7.4/7.5) | `config_repl.c:1267` (send fn), `config_repl.c:1308-1381` (handler), `cluster.c:750-780` (primary-side accept/reject branch + term-adoption) |
| `RequestVote`/`PreVote(i,j)` reply — endpoint-failure routing, all timings | A vote/PreVote endpoint failure is always counted through the production tally (`vote_reply_dispatch` / `pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path — true both at connect-time (bug-1423/1424, regression-locked) and post-connect, i.e. a failure after connect but before a reply arrives (raft-adapter-conformance Req 4.2/4.3, closing that one remaining failure-timing gap). `vote_ep_err_handler` (real-vote) and `pre_vote_ep_err_handler` (PreVote) are separate handlers sharing one `CTRL_PRIMARY`-cleanup helper | `ha.c:7667` (`vote_ep_err_handler`), `ha.c:7705` (`pre_vote_ep_err_handler`), `ha.c:7627` (shared cleanup helper), registered at `ha.c:692` / `ha.c:831` |
| `ClientRequest(i,v)` | Job submit → WAL append → `repl_wait_majority_ack` | `persist.c`, `repl.c` |
| `UpdateTerm(i,j,m)` | Term CAS in `handle_request_vote` / vote reply handler | `ha.c:1218-1230` |
> Full gap-analysis citations, options considered, and design rationale for the `raft-adapter-conformance` rows above (including the dormant-sender decision and the vote/PreVote handler-split decision): `.kiro/specs/raft-adapter-conformance/research.md` and `design.md`. This table stays the terse, current pointer; the evidentiary trail lives there.
### Safety Invariants Verified
**Election Safety** (≤1 leader per term): Enforced by `votedFor` — a server grants at most one vote per term (`raft_on_vote_request` line 145-151). A candidate needs majority votes; each server votes at most once per term.
@ -107,6 +113,9 @@ When modifying HA protocol code (election, replication, heartbeat), verify:
- [ ] Durability flushing makes a contiguous prefix durable; `durable_lsn` is a prefix watermark (never a per-entry set)
- [ ] Causal dependencies are appended in order (cause before effect, e.g. submit before start)
- [x] (post `ha-commit-durability`, landed 2026-06-18) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority
- [x] (post `raft-adapter-conformance`, landed 2026-07-29) every reject of a replication-class message (WAL catch-up entry, configuration push) sends an explicit reply distinguishable from a lost message — never a silent drop (`ha_send_wal_ack_reply_ex(success=0)` for WAL catch-up, bug-1479; `send_config_push_reply(accepted=0)` for config-push, Req 2.2-2.5)
- [x] (post `raft-adapter-conformance`, landed 2026-07-29) every vote/PreVote endpoint failure — connect-time or post-connect (a failure after connect but before a reply) — is counted through the production tally (`vote_reply_dispatch`/`pre_vote_reply_dispatch`), never the legacy `handle_request_vote_reply` path (Req 4.1-4.4)
- [x] (post `raft-adapter-conformance`, landed 2026-07-29) `raft_send_append_entries`/`ha_cb_send_append_entries_stub` remains dormant in production (zero callers) by deliberate documented decision, not oversight — `ha_send_append_entries_stub_call_count` stays 0 while `repl_broadcast`/`ha_stream_wal_to_standby` fire for the same scenario (Req 3.3)
## HA Design Constraints

81
src/controller/cluster.c
View file

@ -669,10 +669,30 @@ static int myreq_handler(vtask_t* task)
break;
case OPCode_upgradeConfig:
#if CONF_ENABLE_UPGRADE_CONFIG
/* OPCode_upgradeConfig is sent BY the primary TO standbys to push config.
* Standbys must apply it; only a primary sending this to itself is invalid. */
if (ha_is_primary(&c->ha)) {
/* Primary should not receive its own config push — ignore */
/* OPCode_upgradeConfig is sent BY an admin client (or xceadmin's
* config-push tool) TO the primary controller, which decodes the
* blob, verifies it, and drives it through config_commit_change()
* (the (term,lsn) marker consensus path) — see
* do_upgrade_config_with_bytes's doc comment, config_repl.c. Only the
* primary can commit a WAL entry, so a standby cannot service this
* request; ha_dispatch_gate already denies OPCode_upgradeConfig for
* every non-primary role (it is absent from the CTRL_STANDBY /
* CTRL_INIT / CTRL_ELECTING allow-lists, ha.c), so this check is a
* defensive belt-and-suspenders guard, not the primary gate.
*
* bug-fix (2026-07-25): this check was inverted from
* commit fd93997f ("election-iomgr-refactor", 2026-05-31) onward —
* it rejected the request WHEN primary, unconditionally reachable
* only on primary (per the dispatch-gate note above), making
* handle_upgrade_config() permanently unreachable from any network
* caller (only unit tests calling it directly ever exercised it).
* Independently diagnosed and documented as a blocking risk by the
* admin-config-tool spec (design.md "Boundary Commitments" /
* "Risks") before config-upgrade-snapshot task 8.6 needed a live
* OP_upgradeConfig sender and hit the same wall. Restored to the
* pre-regression (correct) polarity. */
if (!ha_is_primary(&c->ha)) {
/* Standby cannot commit a config change — reject loud. */
xce_reply_t ureply;
init_reply_by_req(&ureply, &request);
ureply.status = -EAGAIN;
@ -729,6 +749,26 @@ static int myreq_handler(vtask_t* task)
&& workreply->data.replconfigpushreplymsg) {
repl_config_push_reply_t *cack =
workreply->data.replconfigpushreplymsg;
/* raft-adapter-conformance Req 7.4/7.5: a config-push reply now
* carries the replying member's current term (task 1 schema
* addition + task 2.1 send side) -- route it through the same
* term-discovery entry point the heartbeat-reply and WAL-NACK
* vectors already use (bug-1421; ha.c raft_lock-guarded
* raft_on_higher_term_observed calls) so a stale-term primary
* adopts the higher term and steps down instead of retrying
* forever against a standby that has already moved on. Applies
* regardless of accept/reject below -- a reject can still carry
* a useful higher term. raft_on_higher_term_observed is itself
* strictly-greater-than-only (ha-async-io.md Heartbeat Model),
* so no separate `>` comparison is needed here; raft_lock
* discipline mirrors the vote-request/pre-vote-request branches
* above (bug-1435 -- serialize against concurrent raft_state_t
* access). */
pthread_mutex_lock(&c->ha.raft_lock);
(void)raft_on_higher_term_observed(&c->ha.raft, cack->term);
pthread_mutex_unlock(&c->ha.raft_lock);
if (cack->accepted) {
repl_on_config_ack(&c->persist.repl, ep, cack->version_tag);
TRACE("myreq_handler: config ACK from inbound standby"
@ -759,6 +799,39 @@ static int myreq_handler(vtask_t* task)
return rc;
}
#ifdef XCE_TESTING
/*
* myreq_handler_for_test: expose myreq_handler for tests (peer_ctrl_msg_
* handler_for_test precedent, ha.c). Builds the vtask_t/vtask_data_t wrapper
* vtask_entry constructs internally in production so a test can decode+
* dispatch a real packet through the actual switch -- including the
* ReplConfigPushReply branch's term-adoption check (raft-adapter-
* conformance Req 7.4/7.5). Packet ownership follows myreq_handler's own
* contract: myreq_handler unconditionally calls xce_decode(pkt, ...) as its
* first step, and xce_decode() itself consumes pkt -- it calls
* free_packet(pkt) (frees pkt->data) and FREEUP(pkt) (frees the io_packet_t
* struct) before returning (request.c). Ownership of `pkt` is therefore
* TRANSFERRED into this call: the caller must NOT touch or free `pkt` after
* calling myreq_handler_for_test -- doing so is a use-after-free/double-free.
* (xce_free_request separately releases the decoded xce_request_t, an
* unrelated stack-local object in myreq_handler.)
*/
int myreq_handler_for_test(endpoint_t* ep, io_packet_t* pkt)
{
vtask_data_t data;
memset(&data, 0, sizeof(data));
data.vtype = VTASK_IO;
data.u.iodata.ep = ep;
data.u.iodata.pkt = pkt;
vtask_t task;
memset(&task, 0, sizeof(task));
task.data = &data;
return myreq_handler(&task);
}
#endif /* XCE_TESTING */
static int vtask_entry(vtask_t* task)
{
int rc;

291
src/controller/config_repl.c
View file

@ -611,10 +611,38 @@ static int load_cluster_config_from_file(char *file, cluster_config_t **config)
/* ── load_primary_config ────────────────────────── */
#ifdef XCE_TESTING
/* load_primary_config_race_hook: test-only synchronization seam (same
* pattern as push_thread_send_hook / xce_wal_stream_request_hook). When
* non-NULL, load_primary_config() calls this immediately after capturing the
* OLD myconf->cluster_config pointer this invocation is about to retire
* (free), right before it actually retires it -- letting a unit test
* deterministically force a SECOND, real, concurrent load_primary_config
* call through this exact window instead of depending on scheduler timing.
* NULL in production -- no behaviour change. See
* test_bug1476_concurrent_load_primary_config_no_uaf_or_double_free
* (test_config_repl.c) for the RED-phase reproduction this was built for:
* with the lock below removed, this hook deterministically reproduces the
* live ASan heap-use-after-free bug-1476 caught (two concurrent
* load_primary_config calls racing the free of the SAME myconf->
* cluster_config pointer). With the lock, a second invocation can never
* even reach this hook while the first is parked here -- it blocks
* acquiring myconf->lock before getting this far, which is the
* serialization proof the fix requires. */
void (*load_primary_config_race_hook)(void) = NULL;
#endif
int load_primary_config(cluster_t *c)
{
myconf_t *myconf = c->myconf;
/* bug-1476: serialize the whole free-decode-publish sequence against a
* concurrent load_primary_config() call for the SAME myconf_t. Held
* across every return path below, including the success path — a
* second caller must block here before it can even reach the race hook
* fired near the retire step, not merely around that step alone. */
mylock_lock(&myconf->lock);
/* Build conf dir path: <workdir>/<name> */
char conf_dir[4096];
snprintf(conf_dir, sizeof(conf_dir), "%s/%s",
@ -627,14 +655,17 @@ int load_primary_config(cluster_t *c)
INFO("No config at '%s/conf/current' (rc=%d) — "
"will fetch from primary after election",
conf_dir, rc);
mylock_unlock(&myconf->lock);
return -ENOENT;
}
/* Heap-allocate conffile so it survives past this stack frame */
free(myconf->conffile);
myconf->conffile = NULL;
if (asprintf(&myconf->conffile, "%s/conf/%s", conf_dir, fname) < 0)
if (asprintf(&myconf->conffile, "%s/conf/%s", conf_dir, fname) < 0) {
mylock_unlock(&myconf->lock);
return -ENOMEM;
}
/* Read raw bytes and check size */
uint8_t *bytes = NULL;
@ -644,6 +675,7 @@ int load_primary_config(cluster_t *c)
ERROR("Primary config '%s' unreadable: %d", myconf->conffile, rc);
free(myconf->conffile);
myconf->conffile = NULL;
mylock_unlock(&myconf->lock);
return rc;
}
@ -653,53 +685,61 @@ int load_primary_config(cluster_t *c)
memcpy(&tag, myconf->config_sha256, 8);
myconf->config_version_tag = be64toh(tag);
/* Free any prior decoded config before overwriting the pointer */
if (myconf->cluster_config) {
free_ClusterConfig(myconf->cluster_config);
free(myconf->cluster_config);
myconf->cluster_config = NULL;
}
/* Decode Cap'n Proto directly into myconf->cluster_config */
rc = load_cluster_config_from_file(myconf->conffile,
&myconf->cluster_config);
/* Decode Cap'n Proto into a LOCAL temporary first (bug-1476 fix part 2):
* decode_ClusterConfig_ptr calloc's and publishes its out-parameter
* pointer BEFORE decode_ClusterConfig populates the nested fields, so
* decoding directly into the shared myconf->cluster_config field would
* expose any other reader of that field to a live-but-incomplete
* object. Only a fully decoded and validated config is ever published
* below. */
cluster_config_t *new_config = NULL;
rc = load_cluster_config_from_file(myconf->conffile, &new_config);
if (rc != 0) {
ERROR("Primary config '%s' failed Cap'n Proto decode: %d",
myconf->conffile, rc);
free(bytes);
free(myconf->conffile);
myconf->conffile = NULL;
mylock_unlock(&myconf->lock);
return rc;
}
if (myconf->cluster_config == NULL) {
if (new_config == NULL) {
ERROR("Primary config '%s' decoded to NULL", myconf->conffile);
free(bytes);
free(myconf->conffile);
myconf->conffile = NULL;
mylock_unlock(&myconf->lock);
return -EINVAL;
}
free(bytes);
fill_default_cluster_config(myconf->cluster_config);
fill_default_cluster_config(new_config);
/* Check for reserved lost_and_found name */
if (config_contains_reserved_name(myconf->cluster_config)) {
if (config_contains_reserved_name(new_config)) {
ERROR("Config '%s' contains reserved name 'lost_and_found'",
myconf->conffile);
free_ClusterConfig(myconf->cluster_config);
free(myconf->cluster_config);
myconf->cluster_config = NULL;
free(myconf->conffile);
myconf->conffile = NULL;
free_ClusterConfig(new_config);
free(new_config);
mylock_unlock(&myconf->lock);
return -EINVAL;
}
/* Default generation to 1 if not set */
if (myconf->cluster_config->config_generation == 0)
myconf->cluster_config->config_generation = 1;
myconf->config_generation = myconf->cluster_config->config_generation;
if (new_config->config_generation == 0)
new_config->config_generation = 1;
myconf->config_generation = new_config->config_generation;
/* Publish: retire the prior config only now that the new one is fully
* decoded and validated — myconf->cluster_config never points at a
* freed or partially-built object. */
if (myconf->cluster_config) {
#ifdef XCE_TESTING
if (load_primary_config_race_hook)
load_primary_config_race_hook();
#endif
free_ClusterConfig(myconf->cluster_config);
free(myconf->cluster_config);
}
myconf->cluster_config = new_config;
/* Check staleness */
myconf->conf_stale = conf_check_stale(conf_dir, myconf->conf_stale_timeout);
@ -709,6 +749,7 @@ int load_primary_config(cluster_t *c)
INFO("Primary config loaded: tag=%016" PRIx64 " gen=%" PRIu64,
myconf->config_version_tag, myconf->config_generation);
mylock_unlock(&myconf->lock);
return 0;
}
@ -1206,6 +1247,50 @@ int handle_config_request(cluster_t *c, endpoint_t *ep, xce_request_t *req)
return rc2;
}
#ifdef XCE_TESTING
void (*config_push_reply_send_hook)(uint64_t term, uint64_t version_tag,
int accepted) = NULL;
#endif
/* send_config_push_reply: construct and send ReplConfigPushReply{accepted,
* version_tag, term} back to the primary over ep. Mirrors
* ha_send_wal_ack_reply_ex (ha.c, bug-1479) exactly: when
* config_push_reply_send_hook is set (XCE_TESTING), the hook is called with
* the built {term, version_tag, accepted} INSTEAD of sending over ep -- lets
* a unit test observe the reply payload without a live endpoint.
* version_tag is always the standby's OWN current config_version_tag (per
* repl_config_push_reply_s's documented semantics, "standby's current
* version tag after decision") -- not the pushed request's tag, so a reject
* correctly reports the unchanged local state. raft-adapter-conformance
* task 2.1 (Req 2.2, 2.3, 2.5, 7.4): every handle_repl_config_push outcome
* -- accept or reject -- sends exactly one explicit reply, never silence. */
static void send_config_push_reply(cluster_t *c, endpoint_t *ep,
xce_request_t *req, int accepted)
{
uint64_t term = (uint64_t)atomic_load_explicit(&c->ha.term,
memory_order_acquire);
uint64_t version_tag = c->myconf->config_version_tag;
#ifdef XCE_TESTING
if (config_push_reply_send_hook) {
config_push_reply_send_hook(term, version_tag, accepted);
return;
}
#endif
repl_config_push_reply_t reply_body = {
.accepted = accepted,
.version_tag = version_tag,
.term = term,
};
xce_reply_t reply;
init_reply_by_req(&reply, req);
reply.status = 0;
reply.data_which = RequestReply_data_replconfigpushreplymsg;
reply.data.replconfigpushreplymsg = &reply_body;
xce_reply(ep, c->myhost, &reply);
}
/* ── handle_repl_config_push ────────────────────────────────────────────────
*
* Standby handler for OPCode_replConfigPush sent by the primary.
@ -1215,13 +1300,17 @@ int handle_config_request(cluster_t *c, endpoint_t *ep, xce_request_t *req)
* 2. Verify SHA-256 integrity
* 3. Save to disk via conf_save_config
* 4. Reload cluster_config in memory via load_primary_config
* 5. Send ReplConfigPushReply with accepted=1
* 5. Send ReplConfigPushReply -- accepted=1 on success (including
* activation-deferred EAGAIN), accepted=0 on any reject (raft-adapter-
* conformance Req 2.2, 2.3, 2.5); every path sends exactly one reply,
* never silence.
*/
int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
{
if (req->body_which != Request_body_replconfigpushrequestmsg
|| !req->body.replconfigpushrequestmsg) {
WARN("handle_repl_config_push: malformed request");
send_config_push_reply(c, ep, req, /*accepted*/0);
return -EINVAL;
}
@ -1241,6 +1330,7 @@ int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
if (!rcpr->raw_bytes || rcpr->raw_bytes_len == 0 || !rcpr->raw_sha256) {
WARN("handle_repl_config_push: missing bytes or sha256 (tag=%016" PRIx64 ")",
rcpr->version_tag);
send_config_push_reply(c, ep, req, /*accepted*/0);
return -EINVAL;
}
@ -1249,6 +1339,7 @@ int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
SHA256(rcpr->raw_bytes, rcpr->raw_bytes_len, computed);
if (memcmp(computed, rcpr->raw_sha256, 32) != 0) {
WARN("handle_repl_config_push: SHA-256 mismatch — rejecting push");
send_config_push_reply(c, ep, req, /*accepted*/0);
return -EBADMSG;
}
@ -1264,6 +1355,7 @@ int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
rcpr->raw_sha256);
if (rc != 0) {
WARN("handle_repl_config_push: conf_save_config failed: %d", rc);
send_config_push_reply(c, ep, req, /*accepted*/0);
return rc;
}
@ -1281,17 +1373,10 @@ int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
if (act_rc != 0 && act_rc != -EAGAIN)
WARN("handle_repl_config_push: config_activate_committed rc=%d", act_rc);
/* ACK the primary */
repl_config_push_reply_t reply_body = {
.accepted = 1,
.version_tag = c->myconf->config_version_tag,
};
xce_reply_t reply;
init_reply_by_req(&reply, req);
reply.status = 0;
reply.data_which = RequestReply_data_replconfigpushreplymsg;
reply.data.replconfigpushreplymsg = &reply_body;
xce_reply(ep, c->myhost, &reply);
/* ACK the primary. -EAGAIN (activation deferred) is NOT a rejection --
* the bytes were saved out-of-band successfully, so this still replies
* accepted=1 (raft-adapter-conformance Req 2.5, 7.5: unchanged). */
send_config_push_reply(c, ep, req, /*accepted*/1);
return 0;
}
@ -1570,6 +1655,21 @@ int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
/* ── handle_upgrade_config ──────────────────────── */
/* Test-only seam (config-upgrade-snapshot task 7): fires in
* do_upgrade_config_with_bytes right after config_commit_change() succeeds
* -- the configuration-change marker has just reached the committed
* prefix, but materialization on every member (apply_committed_entry's
* WAL_CONFIG_MARKER case, persist.c) has not necessarily completed
* anywhere yet. This is exactly the window a later regression test needs a
* deterministic way to land a forced leadership change in (Requirements
* 4.3, 5.4, 6.3). NULL in production -- no behaviour change. Mirrors
* xce_snapshot_chunk_hook's shape. Declared extern in the test file (same
* pattern as xce_snapshot_chunk_hook). */
#ifdef XCE_TESTING
int (*xce_upgrade_marker_committed_hook)(cluster_t *c,
uint64_t version_tag) = NULL;
#endif
/*
* do_upgrade_config_with_bytes: core upgrade logic called after the raw
* capnp bytes have been extracted from the wire via
@ -1586,7 +1686,12 @@ int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
* 7. Push to standbys and wait for the majority-ACK barrier; on failure
* reply with send_upgrade_error and return (no success ACK, no restart)
* 8. ACK admin tool (success ⇔ majority-acked config)
* 9. Wait for in-flight checkpoint, then signal restart
*
* The function's job ends at step 8. Materialization of the new
* configuration on every member -- including this one -- happens
* asynchronously and independently via each member's own apply of the
* committed WAL_CONFIG_MARKER (apply_committed_entry, persist.c,
* config-upgrade-snapshot task 5). No restart is requested or required.
*/
static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
xce_request_t *req,
@ -1720,6 +1825,11 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
return rc;
}
#ifdef XCE_TESTING
if (xce_upgrade_marker_committed_hook)
(void)xce_upgrade_marker_committed_hook(c, c->myconf->config_version_tag);
#endif
/* 8. ACK the admin tool (success ⇔ majority-acked config) */
upgrade_config_reply_t urep;
memset(&urep, 0, sizeof(urep));
@ -1734,41 +1844,13 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
reply.data.upgradeconfigreplymsg = &urep;
xce_reply(ep, c->myhost, &reply);
/* NOTICE (bug-857, spec `.kiro/specs/config-upgrade-snapshot/`) — config
* upgrade MUST snapshot current state, not re-derive it per node.
* Today the upgrade commits the config-change marker, then each member
* RESTARTS and re-materializes its post-upgrade state by replaying its own
* pre-upgrade WAL under the NEW config (steps 9 + restart recovery). Replay
* is config-dependent (`lost_and_found` re-homes entities whose config
* entity vanished), so members can DIVERGE in materialized state after an
* upgrade. The fix: at the config-change marker's (term,lsn) boundary the
* controller must materialize a full ENTITY-BEARING state snapshot, embed
* the new config in it, and install that SAME snapshot on every member via
* the snapshot_transfer_s transport (install-before-activate, hash-verified)
* so post-upgrade state is identical by construction and no member
* re-derives it. HARD dependency: `entity-bearing-snapshots` (snapshots are
* a 16-byte LSN sentinel today; the entity payload does not exist yet).
* Until that lands, the per-node-replay path below is the interim behavior. */
/* 9. Wait for in-flight checkpoint (up to 30 s), then signal restart */
struct timespec ckpt_start, ckpt_now;
clock_gettime(CLOCK_MONOTONIC, &ckpt_start);
while (atomic_load(&c->persist.checkpoint_in_progress)) {
usleep(10000);
clock_gettime(CLOCK_MONOTONIC, &ckpt_now);
if (ckpt_now.tv_sec - ckpt_start.tv_sec > 30) {
ERROR("Checkpoint did not complete within 30 s"
" -- restarting anyway");
break;
}
}
/* Materialization now happens via apply_committed_entry's
* WAL_CONFIG_MARKER case (persist.c, config-upgrade-snapshot task 5) on
* every member's own apply of this marker -- no restart needed. */
INFO("Config upgraded to tag=%016" PRIx64 " gen=%" PRIu64
" (acked=%d standbys) -- requesting restart",
" (acked=%d standbys)",
c->myconf->config_version_tag, new_gen, acked);
atomic_store(&c->restart_requested, 1);
int stop_val = 1;
cbuf_put(&c->wbuf, &stop_val);
return 0;
}
@ -2359,19 +2441,50 @@ int ha_stream_wal_to_standby(endpoint_t *ep, const char *wal_path,
* never sent on the live broadcast path either
* (persist_wal_write_replicated rejects type==WAL_CHECKPOINT with
* -EINVAL). Skip it entirely -- do not build/send a catchup entry
* and do not advance prev_streamed_lsn/prev_streamed_term, so the
* NEXT real entry's AppendEntries prev tuple chains to the last
* REAL entry before this gap, exactly matching what a standby that
* never records a checkpoint marker in its own log expects.
* Without this, a standby receiving one gets stuck retrying the
* same LSN forever (the receive-side guard correctly refuses it). */
if (hdr.type == WAL_CHECKPOINT) {
free(locked_pbuf);
off += (uint32_t)(sizeof(hdr) + hdr.payload_len);
continue;
}
* for it.
*
* bug-1475 (fix): DO fall through to the SAME unconditional
* prev-tracker advance every other scanned entry gets below
* (the "for EVERY scanned entry" logic a few lines down) instead
* of special-casing a skip here. A standby's own current_lsn/
* durable_floor ALSO advances past this exact LSN the moment it
* independently applies the SAME committed WAL_CONFIG_MARKER (any
* member that commits a marker immediately writes its own local
* checkpoint at the marker's own LSN+1, deterministically the
* SAME lsn on every member that has caught up through the
* marker) -- so the checkpoint's LSN is a real, held position on
* BOTH sides, not a gap. The OLD behavior (never advancing the
* tracker across it) under-counted the sender's prev_log_index by
* one for every checkpoint encountered, permanently diverging it
* from the receiver's own (now correctly co-tracked -- see
* ha_apply_term_entry_via_library's bug-1475 fix) log_len: the
* first post-checkpoint entry would spuriously "succeed" at the
* WRONG index, after which every following entry's prev_log_index
* permanently exceeded the receiver's true log length --
* ae_reject_logmismatch forever, with the primary re-streaming
* the identical doomed range every heartbeat. Falling through
* here (falling to the same code path as every other entry)
* fixes the divergence at its source.
*
* bug-1475 follow-up: this tracker advance alone was NOT
* sufficient — persist_entry_term(p, checkpoint_lsn) (the term
* this tracker reads for prev_log_term) turned out to be
* TIME-DEPENDENT before a further persist.c fix: it correctly
* read absent (0) right after the checkpoint, but silently
* flipped to WAL_LEGACY_TERM (1) the instant this leader
* appended any LATER real entry (which has already happened by
* the time this streamer runs), because persist.c's term-index
* array growth zero-filled the checkpoint's never-explicitly-
* recorded gap slot and read that zero-fill back as present. A
* receiver that had not yet appended past its own checkpoint
* still (correctly) read 0 for the same LSN, so logOk's
* prev_log_term equality check mismatched (0 != 1) and rejected
* forever. Fixed with a TERM_INDEX_UNSET sentinel in persist.c
* (record_entry_term/entry_term_locked) that keeps a
* never-recorded slot reading absent permanently, independent of
* later growth — see TERM_INDEX_UNSET's comment there. */
if (hdr.lsn > standby_lsn) {
if (hdr.lsn > standby_lsn && hdr.type != WAL_CHECKPOINT) {
wal_catchup_entry_t *entry = calloc(1, sizeof(*entry));
if (!entry) {
free(locked_pbuf);
@ -2543,8 +2656,14 @@ static int repl_ep_still_registered(replicator_t *r, repl_ep_t ep)
return found;
}
/* re-derive the replication target exactly as ha.c:2771-2776 / D6:
* target = max(committed_lsn, last_written), last_written = current_lsn - 1. */
/* re-derive the replication target via the SAME pure function
* repl_ack_decode_and_dispatch (ha.c) uses, so this continuation path and
* the heartbeat decision path can never disagree on what a standby must
* catch up to (bug-1478: this function used to duplicate the raw
* max(committed_lsn, last_written) formula inline, independently of the
* ha.c call site the fix landed in — a local-only, never-replicated
* WAL_CHECKPOINT trailing an already-advancing committed_lsn permanently
* wedged catch-up here as well, via the exact same mechanism). */
static uint64_t catchup_derive_target(cluster_t *c)
{
uint64_t committed = atomic_load_explicit(&c->persist.committed_lsn,
@ -2554,7 +2673,7 @@ static uint64_t catchup_derive_target(cluster_t *c)
? c->persist.current_lsn - 1
: 0;
pthread_mutex_unlock(&c->persist.lock);
return last_written > committed ? last_written : committed;
return ha_compute_repl_target_lsn(committed, last_written);
}
/* Invoke the catch-up streaming path for [standby_lsn+1, target]. Mirrors the

23
src/controller/config_repl.h
View file

@ -453,6 +453,29 @@ extern int (*xce_probe_config_tag_hook)(const char *host, uint16_t port,
*/
int load_primary_config_for_test(cluster_t *c);
/*
* load_primary_config_race_hook: test-only synchronization seam. See the
* doc comment at its definition (config_repl.c) for the exact semantics —
* it lets a unit test deterministically force two concurrent
* load_primary_config() calls for the SAME myconf_t through the race window
* bug-1476's live ASan heap-use-after-free crossed. Set by test
* infrastructure (test_config_repl.c) before the relevant tests run.
*/
extern void (*load_primary_config_race_hook)(void);
/*
* config_push_reply_send_hook: test-only injection point for the outbound
* ReplConfigPushReply built by send_config_push_reply() (config_repl.c).
* When non-NULL, called with (term, version_tag, accepted) INSTEAD of
* building/sending the real reply over ep -- lets a unit test observe a
* config-push accept OR reject reply without a live endpoint
* (raft-adapter-conformance task 2.1, mirrors ha_wal_ack_send_hook /
* bug-1479). NULL in production. Set by test infrastructure
* (test_config_repl.c) before the relevant tests run.
*/
extern void (*config_push_reply_send_hook)(uint64_t term, uint64_t version_tag,
int accepted);
/*
* push_ctx_t: exposed under XCE_TESTING so unit tests can construct
* synthetic push contexts to drive wait_all_with_timeout_for_test().

1063
src/controller/ha.c
View file

File diff suppressed because it is too large Load diff

280
src/controller/ha.h
View file

@ -320,6 +320,44 @@ typedef struct {
* cleared and the config is activated.
* NEVER self-terminates (loud non-serving
* fence only). 0 at init (ha_init memset). */
/* ── New field added by config-upgrade-snapshot spec (Task 6.1) ────── */
_Atomic int upgrade_materialization_fence; /* Upgrade-snapshot
* materialization-and-hash-verify
* fence (Req 1.8, 3.2, 3.4) — a NEW,
* INDEPENDENT fence, deliberately NOT
* merged into config_serve_fence
* above: that fence tracks whether the
* out-of-band config-PUSH bytes match
* the committed marker; THIS fence
* tracks whether THIS MEMBER'S OWN
* upgrade-snapshot materialization
* (the marker-triggered checkpoint,
* config-upgrade-snapshot task 5) has
* actually completed for the CURRENT
* committed marker, with the live
* on-disk config bytes still
* hash-verifying against it right
* now. These are logically distinct
* and can disagree in either
* direction (e.g. bytes can already
* hash-match — config_serve_fence
* clear — while this member's own
* checkpoint for that exact marker
* has not run yet, immediately after
* a marker commits but before its
* forced checkpoint fires). Set by
* upgrade_materialization_verify()
* on materialization-incomplete or a
* hash mismatch against a FRESH
* re-read of the live config bytes
* (never a cached/embedded copy);
* cleared on success. While set, the
* member MUST NOT serve under the new
* configuration or become primary; it
* STAYS ALIVE and keeps retrying.
* NEVER self-terminates (loud
* non-serving fence only). 0 at init
* (ha_init memset). */
/* ── New field added by raft-drive-async spec (Task 2.1) ──────── */
_Atomic int reconnect_inflight[MAX_HA_CANDIDATES]; /* Per-peer
* single-flight guard for the offloaded
@ -558,6 +596,32 @@ int ha_primary_log_append(cluster_t *c, uint32_t type,
int ha_raft_on_repl_ack(cluster_t *c, int abs_peer_idx, uint64_t acked_lsn,
uint64_t reply_term);
/*
* ha_raft_on_repl_nack (bug-1479): drive the library on a REJECTED
* AppendEntries (WalAckReply.success == 0, i.e. the standby's log conflicts
* at the sent prevLogIndex/prevLogTerm). Routes the rejection through
* raft_on_append_entries_reply(success=0), which backs off
* next_index[abs_peer_idx] by one (raft.tla's AppendEntries failure branch,
* floored at 1) -- mirrors ha_raft_on_repl_ack's bounds-checked peer-index
* resolution exactly, but does NOT call raft_advance_commit (a rejection
* can never advance commit).
*
* Before this fix, a NACK was only ever used for stale-term discovery
* (raft_on_higher_term_observed) and otherwise silently dropped -- the
* library's own, already-correct nextIndex backoff mechanism was never
* invoked, so a standby stuck in a genuine log conflict (an ordinary
* post-election occurrence) never had its resend position corrected: the
* primary kept broadcasting new entries assuming agreement at an index the
* standby had already rejected, forever.
*
* Returns the corrected resume LSN (next_index[abs_peer_idx] - 1, i.e. the
* position the standby is now believed caught up through) via
* *corrected_lsn_out on success; returns -EINVAL (leaving *corrected_lsn_out
* untouched) on a no-cluster context or a rejected peer index, matching
* ha_raft_on_repl_ack's defense-in-depth bounds check. */
int ha_raft_on_repl_nack(cluster_t *c, int abs_peer_idx, uint64_t reply_term,
uint64_t *corrected_lsn_out);
/*
* ha_repl_ack_abs_peer_idx (ha-term-safety task 3.3 remediation): resolve the
* ABSOLUTE library peer index from an ACK endpoint's stored peer_idx, branching
@ -755,6 +819,122 @@ int ha_dispatch_gate(ha_state_t *ha, int opcode);
*/
int ha_config_promotion_fenced(const ha_state_t *ha);
/*
* upgrade_materialization_verify (config-upgrade-snapshot task 6.1, Req 1.8,
* 3.2, 3.4): the Activation Fence's own verify function. Mirrors
* config_activate_committed's exact shape (config_repl.c:304-377) for a
* DIFFERENT, independently-tracked condition — see ha_state_t's
* upgrade_materialization_fence field doc comment above for the full
* rationale on why this must be a separate fence from config_serve_fence.
*
* On every call: reads the current committed WAL_CONFIG_MARKER (the
* authority for the target upgrade), checks whether THIS MEMBER'S OWN
* checkpoint machinery has already captured a materialization reference for
* that EXACT marker (c->persist.cfgref_present/cfgref_tag/cfgref_sha,
* stamped by every checkpoint via persist.c's scan_latest_committed_config_
* ref), and separately re-reads the LIVE, currently-active config bytes off
* disk FRESH (never a cached/embedded copy) to hash-verify them against the
* marker too. Both must hold for the fence to clear.
*
* Returns 0 (fence cleared or absent-marker benign case) on success;
* -EAGAIN when materialization is incomplete or the hash mismatches (fence
* SET, logged loud only on the 0->1 transition); -EINVAL/-EIO on a genuine
* error reading the committed marker. NEVER exits/aborts on this condition
* (controller/CLAUDE.md never-self-terminate invariant) — always fail loud +
* fence + stay alive + retriable.
*/
int upgrade_materialization_verify(cluster_t *c);
/*
* upgrade_materialization_fenced: 1 when this member's own upgrade-snapshot
* materialization for the current committed config marker is incomplete or
* hash-mismatched (upgrade_materialization_fence set); 0 otherwise. Trivial
* atomic-read accessor — mirrors config_serve_fenced (config_repl.c:262-268)
* exactly, for the new independent fence. Pure atomic read (no I/O).
*/
int upgrade_materialization_fenced(const cluster_t *c);
/*
* upgrade_materialization_retry_fn (config-upgrade-snapshot task 6.2, Req
* 3.4, 5.5): re-runs upgrade_materialization_verify so a member whose
* materialization was incomplete or hash-mismatched re-checks and clears the
* fence once conditions allow. Does the actual blocking disk I/O; production
* callers invoke it off the delaymgr thread via
* schedule_upgrade_materialization_retry's trampoline (below) — mirrors
* config_activate_retry_fn (config_repl.h). Public so tests can drive it
* directly without a vtask pool. `arg` is a `cluster_t *`. Returns 0 on
* success/already-unfenced, -EAGAIN while still fenced, negative errno on a
* genuine error.
*/
int upgrade_materialization_retry_fn(void *arg);
/*
* schedule_upgrade_materialization_retry (config-upgrade-snapshot task 6.2,
* Req 3.4, 5.5): non-blocking, self-triggering retry for the Activation
* Fence — mirrors config_schedule_activate_retry's exact delaymgr-
* trampoline-then-vtask shape (config_repl.c:409-507): a one-shot delaymgr
* task (NON-BLOCKING — only enqueues) offloads the actual blocking re-verify
* work (upgrade_materialization_verify's disk reads) onto a VTASK_LIVE_BLOCK
* vtask so the delaymgr thread itself never blocks (ha-async-io.md
* thread-role contract). No-op when the member is NOT upgrade-materialization
* fenced (steady state schedules nothing — no busy-spin) or when the
* delaymgr is not running (unit-test fixture path; caller drives the retry
* directly instead). Falls back to running the verify inline (stay alive,
* never silently drop a scheduled retry) if the vtask pool is unavailable.
* Returns 0 on enqueue/no-op, negative on enqueue failure.
*/
int schedule_upgrade_materialization_retry(cluster_t *c);
/*
* ha_phase2_retry_fn (bug-1477): re-attempts cluster_init_phase2 for a node
* whose earlier attempt(s) failed and never set ha->phase2_done. Idempotent
* -- a no-op once phase2_done is already set by ANY call site (the shared
* once-per-process contract every cluster_init_phase2 caller relies on).
* Does the actual blocking work (cluster_init_phase2 runs persist_recover,
* wires the commit engine, etc.); production callers invoke it off the
* delaymgr thread via ha_schedule_phase2_retry's trampoline (below) --
* mirrors upgrade_materialization_retry_fn/config_activate_retry_fn exactly.
* Public so tests can drive it directly without a vtask pool. `arg` is a
* `cluster_t *`. Returns 0 on success/already-done, the underlying negative
* errno on a genuine cluster_init_phase2 failure (and self-reschedules
* before returning, so the caller need not).
*
* Root cause this closes: a node that takes myapp_run's skip_election
* passive path (bug-849) -- because a leader was already known after the
* startup stagger -- makes exactly ONE cluster_init_phase2 attempt. If that
* attempt races config availability and fails, NOTHING else ever retries
* it: raft_cb_become_follower / raft_cb_become_leader /
* ha_elect_evaluate_and_signal (the other three call sites) only fire via
* THIS node's own raft-library state transitions, which never happen for a
* node that stays passively CTRL_STANDBY while its primary keeps
* heartbeating normally. The node is then permanently stuck without a
* wired commit engine -- silently unable to apply ANY committed entry for
* the rest of its process lifetime (never self-terminates, per
* controller/CLAUDE.md's Process Lifecycle invariant, but never recovers
* either without this retry).
*/
int ha_phase2_retry_fn(void *arg);
/*
* ha_schedule_phase2_retry (bug-1477): non-blocking, self-rescheduling retry
* for a node whose cluster_init_phase2 has not yet completed -- mirrors
* config_schedule_activate_retry / schedule_upgrade_materialization_retry's
* delaymgr-trampoline-then-vtask shape, except it re-arms ITSELF on failure
* (via ha_phase2_retry_fn) since -- unlike those two, which are re-triggered
* externally by every commit-advance/config-push event -- there is no
* other recurring event that would naturally re-trigger this for a node
* stuck on the passive skip_election path. Retries at ha->config.
* heartbeat_interval_ms cadence (the natural rhythm config/connectivity
* changes arrive on), indefinitely (never gives up -- never-self-terminate
* invariant). No-op when phase2_done is already set (steady state
* schedules nothing -- no busy-spin) or when the delaymgr is not running
* (unit-test fixture path; caller drives the retry directly instead).
* Falls back to running the retry inline (stay alive, never silently drop
* a scheduled retry) if the vtask pool is unavailable. Returns 0 on
* enqueue/no-op, negative on enqueue failure.
*/
int ha_schedule_phase2_retry(cluster_t *c);
/*
* ha_apply_wal_catchup_entry: apply a single WAL catch-up entry to local
* persistent storage and advance ha->applied_lsn on success.
@ -763,15 +943,36 @@ int ha_config_promotion_fenced(const ha_state_t *ha);
* On success (rc == 0): applied_lsn advances to entry->lsn+1 and a
* WalAckReply {term, lsn, success=1} is sent back to the primary via
* xce_reply(ep, c->myhost, &reply) (Task 3.1, Req 3.1).
* On error (rc < 0): logs ERROR, returns error; no ACK sent (Task 3.2).
* On a genuine apply/persist ERROR (rc < 0): logs ERROR, returns error; no ACK
* sent at all (Task 3.2) -- there is nothing coherent to report back yet.
* On a term-carrying entry's AppendEntries REJECT (logOk fails -- forward gap
* or a genuine (term,lsn) conflict; rc == 0, entry not accepted): bug-1479 --
* a WalAckReply {term, lsn, success=0} (a NACK) IS now sent, so the primary's
* ha_raft_on_repl_nack can back off nextIndex and immediately mark this
* standby for catch-up. Before this fix the reject was dropped with no reply
* at all, so the primary's NACK-handling could never engage no matter how
* correct it was -- the standby would loop rejecting the identical
* re-broadcast entry forever.
* applied_lsn is in-memory only — not persisted to disk.
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK is skipped.
* ep and c may be NULL in unit-test builds (XCE_TESTING); ACK/NACK is skipped
* (or captured via ha_wal_ack_send_hook when set).
* Requirements: 3.1, 3.2, 3.3, 3.4, 3.5
*/
int ha_apply_wal_catchup_entry(ha_state_t *ha, persist_t *p,
wal_catchup_entry_t *entry,
struct endpoint_t *ep, cluster_t *c);
#ifdef XCE_TESTING
/*
* ha_wal_ack_send_hook: test-only seam for the outbound WalAckReply built by
* ha_send_wal_ack_reply_ex (bug-1479). When non-NULL, called with
* (term, lsn, success) INSTEAD OF building/sending the real reply over ep --
* lets a unit test assert that an AppendEntries reject now sends a NACK
* (success=0), without needing a live endpoint. NULL in production.
*/
extern void (*ha_wal_ack_send_hook)(uint64_t term, uint64_t lsn, int success);
#endif
/* ha_nop_reply_state_report — populate a heartbeat reply with stateReport.
* Fills *sr from ha->applied_lsn (atomic read) and myconf->config_version_tag,
* then sets reply_out->data_which = stateReportMsg and
@ -1023,6 +1224,54 @@ void ha_process_standby_state(ha_state_t *ha, int peer_idx,
uint64_t reported_tag, uint64_t reported_lsn,
uint64_t committed_tag, uint64_t committed_lsn);
/*
* ha_compute_repl_target_lsn: the replication/convergence target used by
* ha_dispatch_repl_action_for_standby's heartbeat-driven decision — normally
* committed_lsn itself, falling back to last_written (the raw locally-
* appended WAL tip) ONLY when committed_lsn is genuinely stuck at 0
* (cluster bootstrap, before any standby has ever acked anything).
*
* bug-1478: the fallback must NOT apply once committed_lsn is already > 0 —
* see ha.c's doc comment on this function for the full mechanism (a
* local-only WAL_CHECKPOINT trailing an already-advancing committed_lsn
* otherwise leaks an unstreamable/unreachable target into every subsequent
* heartbeat decision, permanently wedging catch-up).
*
* Pure, side-effect-free.
*/
uint64_t ha_compute_repl_target_lsn(uint64_t committed_lsn,
uint64_t last_written);
/*
* ha_compute_repl_stream_ceiling_lsn (bug-1479, gap 3): the upper bound of
* what a leader may actually STREAM to a standby via ha_stream_wal_to_standby
* — distinct from ha_compute_repl_target_lsn's CONVERGENCE target above.
*
* Bounding the stream range at committed_lsn is a permanent deadlock whenever
* the very standby being streamed to must ACK before committed_lsn can
* advance at all (e.g. a 3-node cluster with one member down and this
* standby log-conflicted after a chaotic election): it can never receive the
* entries it needs to ack to unblock the majority. raft.tla's real
* AppendEntries replicates the leader's FULL log — leaderCommit is advisory
* to the follower, never a cap on what gets sent. Streaming past commit is
* always safe: a later leadership change's own (term,lsn) conflict-truncate
* (ha_consume_append_entries) correctly unwinds anything that turns out not
* to have been committed.
*
* bug-1478 guard: never widen the ceiling to exactly the trailing local-only
* checkpoint's own consumed LSN slot — no standby's applied_lsn can ever
* legitimately reach that value (WAL_CHECKPOINT is never streamed or applied
* by any standby, by design), so doing so would reintroduce bug-1478's
* permanent-non-convergence trap. Falling back to committed_lsn for just
* that transient instant is safe — the very next real entry appended after
* the checkpoint widens the ceiling normally again.
*
* Pure, side-effect-free — safe to unit test directly.
*/
uint64_t ha_compute_repl_stream_ceiling_lsn(uint64_t committed_lsn,
uint64_t last_written,
uint64_t checkpoint_lsn);
/* ── Task 3.5c: Standby snapshot receive handler ──────────────────────────── */
/*
@ -1173,6 +1422,22 @@ void ha_cb_commit_advanced_for_test(ha_state_t *ha, uint64_t commit_index);
* term via the library. TEST-ONLY.
*/
void ha_arm_own_term_gate_for_test(cluster_t *c);
/*
* ha_send_append_entries_stub_call_count (raft-adapter-conformance task 4.1,
* Requirement 3.3): test-only seam counting invocations of
* ha_cb_send_append_entries_stub, the on_send_append_entries callback. That
* callback is reachable only via the raft library's raft_send_append_entries
* (zero production callers) or ha_cb_send_append_entries_for_test above —
* see the DORMANT IN PRODUCTION comment on ha_cb_send_append_entries_stub in
* ha.c for the full finding. A test that drives only the production
* replication path (repl_broadcast / ha_stream_wal_to_standby) must observe
* this counter stay at 0; a future change that accidentally gives
* raft_send_append_entries a production caller moves it off zero. NULL in
* production (the whole symbol does not exist outside XCE_TESTING).
* TEST-ONLY.
*/
extern uint64_t ha_send_append_entries_stub_call_count;
#endif
/*
@ -1692,8 +1957,15 @@ uint32_t ha_jittered_election_delay(ha_state_t *ha, uint32_t base);
* - HA_REPL_PUSH_CONFIG: In production builds, calls push_config_to_standbys(c, …).
* In XCE_TESTING builds, calls ha_repl_push_config_for_standby_hook(c, peer_idx).
* - HA_REPL_STREAM_WAL: calls ha_stream_wal_to_standby(ep, wal_path, standby_lsn,
* committed_lsn) to batch-stream WAL entries via xce_request() (non-blocking).
* In XCE_TESTING builds, calls ha_repl_stream_wal_hook if non-NULL.
* stream_ceiling_lsn) to batch-stream WAL entries via xce_request()
* (non-blocking). stream_ceiling_lsn (bug-1479 gap 3) is
* ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written,
* checkpoint_lsn) when c is non-NULL — the leader's own log tail, NOT
* just committed_lsn, so a standby whose own ack is required to advance
* committed_lsn can still receive the entries it needs to ack. Falls
* back to committed_lsn when c is NULL (test fixtures with no real
* persist_t). In XCE_TESTING builds, calls ha_repl_stream_wal_hook if
* non-NULL.
* - HA_REPL_SEND_SNAPSHOT: calls ha_send_snapshot_to_standby(ep, snap_path, lsn).
* In XCE_TESTING builds, calls ha_repl_send_snapshot_hook(peer_idx) if non-NULL.
* - HA_REPL_CONVERGED: no-op.

1
src/controller/include/worker.h
View file

@ -13,5 +13,6 @@ extern int init_worker(cluster_t*, endpoint_t*, const char*,
extern int rejoin_worker(cluster_t*, worker_t*, endpoint_t*,
hostattrs_t*, xce_hostconfig_t*);
extern int restore_worker(void*, xce_worker_t);
extern int restore_host_state(cluster_t*, const char*, uint32_t);
extern int worker_from_capnp(cluster_t*, const void*, size_t);
#endif

8
src/controller/main.c
View file

@ -400,6 +400,14 @@ int main(int argc, char* argv[])
myconf.xstrings_delta = arguments.xstrings_delta;
memset(&myconf.peers_cfg, 0, sizeof(myconf.peers_cfg));
/* bug-1476: myconf.lock serializes load_primary_config() against
* concurrent invocations for this myconf_t — must be initialized before
* cluster_run/myapp_run can ever call it. */
if (mylock_init(&myconf.lock) != 0) {
ERROR("Failed to initialize myconf lock");
return -1;
}
if (arguments.peers_str) {
if (xce_parse_peers(arguments.peers_str, &myconf.peers_cfg) != 0) {
fprintf(stderr,

12
src/controller/meson.build
View file

@ -54,11 +54,14 @@ controller = executable('xce_controller',
link_args: wrap_link_args,
install: true)
xce_test_src = [ 'testsuite/test.c', 'testsuite/waldump.c', 'testsuite/walcompare.c', 'testsuite/snapverify.c', 'testsuite/snapdiff.c']
xce_test_src = [ 'testsuite/test.c', 'testsuite/waldump.c', 'testsuite/walcompare.c', 'testsuite/snapverify.c', 'testsuite/snapdiff.c', 'testsuite/configcompare.c']
readline_dep = dependency('readline')
ncurses_dep = dependency('ncurses')
history_dep = dependency('history')
ssl_dep = dependency('openssl') # also needed by configcompare.c (SHA256());
# controller_unit_test_deps below re-assigns
# the same dependency() result, harmless
xce_test_deps = [
libxce_dep,
@ -70,6 +73,7 @@ xce_test_deps = [
tinylog_dep,
zlib_dep, # waldump: crc32 WAL validation
c_capnproto_dep, # waldump: decode JOB_SUBMIT jobid
ssl_dep, # configcompare: SHA256() active-config hashing
]
xce_test = executable('xce_test',
xce_test_src,
@ -157,6 +161,8 @@ controller_unit_test_src = [
'testsuite/snapverify.c',
'testsuite/test_snapdiff.c',
'testsuite/snapdiff.c',
'testsuite/test_configcompare.c',
'testsuite/configcompare.c',
'testsuite/test_persist_snapshot_entities.c',
'testsuite/test_apply_seam.c',
'testsuite/test_apply_committed_entry.c',
@ -168,6 +174,9 @@ controller_unit_test_src = [
'testsuite/test_job_array_apply.c',
'testsuite/test_job_array_fence.c',
'testsuite/test_job_array_replay_fence.c',
'testsuite/test_upgrade_snapshot_restore.c',
'testsuite/test_upgrade_snapshot_apply.c',
'testsuite/test_bug857_regression.c',
'ctrlstatus.c',
'commit.c',
'job.c',
@ -184,6 +193,7 @@ controller_unit_test_src = [
'ctrl_metrics.c',
'config_repl.c',
'slowlane.c',
'cluster.c',
]
ssl_dep = dependency('openssl')

40
src/controller/myapp.c
View file

@ -251,6 +251,39 @@ int myapp_run(myconf_t *myconf)
}
/* Ignore other errors — node may have no config yet (first boot/standby);
* it fetches config via the primary's push post-election (Req 7.3). */
/* config-upgrade-snapshot task 6.2 (Req 5.2): resume Activation-Fence
* verification here too, unconditionally, so a crash/restart resumes
* materialization-verification before serving rather than implicitly
* treating a fresh process's zero-initialized (unfenced) fence as
* "already verified". Sequenced AFTER config_activate_committed
* deliberately: that call is what establishes/repairs which config
* BYTES are locally active for the committed marker; only once that
* has run does the separate, sequential question
* upgrade_materialization_verify asks -- has THIS MEMBER'S OWN
* checkpoint machinery actually materialized under that marker? --
* become meaningful to (re-)evaluate. persist_recover() (step 6,
* above) has already restored c.persist.cfgref_* from the latest
* loaded checkpoint by this point, so the materialization signal
* this reads is accurate here, at boot, same as it would be after
* any later checkpoint. Same "ignore other errors, log loud only on
* the fenced case" treatment as config_activate_committed above --
* ha_dispatch_gate and the promotion gate (ha.c) are what actually
* enforce the fence from here on; this call's only job is to seed
* the correct fence value at boot instead of leaving a freshly-
* started process at its unfenced zero-default regardless of
* whether materialization genuinely completed before the crash. */
int mat_rc = upgrade_materialization_verify(&c);
if (mat_rc == -EAGAIN) {
ERROR("Pre-election upgrade materialization FENCED: this "
"member's own upgrade-snapshot materialization for the "
"committed config marker is incomplete or hash-mismatched "
"— member NON-SERVING and NON-PROMOTABLE, staying alive "
"and retrying (config-upgrade-snapshot R1.8/3.2/3.4/5.2)");
}
/* Ignore other errors/success silently — mirrors the "ignore other
* errors" handling above; ha_dispatch_gate / the promotion gate
* enforce the fence going forward. */
}
/* ── Step 8: ha_start_election_async ─────────────────────────────────── */
@ -387,8 +420,11 @@ skip_election:;
int prc = cluster_init_phase2(&c);
if (prc != 0) {
WARN("myapp_run: cluster_init_phase2 on standby rejoin "
"failed (rc=%d) — WAL not recovered; falling back to "
"election-path recovery", prc);
"failed (rc=%d) — WAL not recovered; member stays "
"alive and will retry (bug-1477: this passive "
"skip_election path never sees the election-result "
"callers that would otherwise retry it)", prc);
ha_schedule_phase2_retry(&c);
} else {
c.ha.phase2_done = 1;
}

1
src/controller/myconf.c
View file

@ -163,4 +163,5 @@ void myconf_fini(myconf_t *myconf)
free(myconf->cluster_config);
myconf->cluster_config = NULL;
}
mylock_fini(&myconf->lock); /* bug-1476 */
}

1146
src/controller/persist.c
View file

File diff suppressed because it is too large Load diff

88
src/controller/repl.c
View file

@ -93,15 +93,60 @@ int repl_add_standby(replicator_t *r, repl_ep_t ep, const char *host, uint32_t p
return -ENOSPC;
}
/* Find first empty slot */
/* Slot allocation (bug-1474 hardening): prefer a genuinely virgin slot
* over one still holding an unexpired bug-1147 disconnect guard
* (last_disconnected_lsn != 0) for a DIFFERENT peer. repl_add_standby
* cannot know whether this call is a legitimate new connection or a
* duplicate/late-joiner reconnect for an already-connected peer (that
* classification defect lives in ha.c, out of scope here) — but either
* way, clobbering an unexpired guard is never correct: it silently
* defeats repl_trim_lsn's WAL retention floor for whichever standby
* the guard was protecting (see .wolf/buglog.json bug-1474).
*
* Pass 1: any slot with ep==NULL && last_disconnected_lsn==0 (virgin —
* never used, or its guard already expired via
* repl_expire_disconnect_slots, which fully zeroes the slot). Always
* safe to take; first such slot by index.
*
* Pass 2 (fallback, only when Pass 1 finds nothing): among slots with
* ep==NULL && last_disconnected_lsn!=0 (guard-holding), reuse the one
* with the OLDEST last_disconnect_time. Deliberate policy: that guard
* is closest to its own natural expiry (repl_expire_disconnect_slots'
* snap_interval_s window), so reusing it discards the LEAST remaining
* WAL-retention protection of any guard-holding slot. With
* MAX_STANDBYS==3 it is possible — under a chaotic multi-disconnect
* scenario with no virgin slot free — for every non-primary slot to be
* guard-holding simultaneously; refusing this connect (-ENOSPC) would
* trade a bounded, already-degraded WAL-retention risk for an outright
* connection failure, which is worse for cluster availability. This
* fallback does NOT touch the -ENOSPC path above, which remains
* reserved for the genuinely-impossible case of more distinct live
* peers than slots exist. */
int idx = -1;
for (int i = 0; i < MAX_STANDBYS; i++) {
if (r->standbys[i].ep == NULL) {
if (r->standbys[i].ep == NULL && r->standbys[i].last_disconnected_lsn == 0) {
idx = i;
break;
}
}
/* Pass 1 found nothing -- about to fall back to Pass 2 (guard-holding
* slot reuse) below. Tracked so the WARN below fires only for the
* genuine fallback case, not the common virgin-slot path. */
int used_fallback_slot = (idx < 0);
if (idx < 0) {
for (int i = 0; i < MAX_STANDBYS; i++) {
if (r->standbys[i].ep != NULL) {
continue;
}
if (idx < 0 ||
r->standbys[i].last_disconnect_time < r->standbys[idx].last_disconnect_time) {
idx = i;
}
}
}
if (idx < 0) {
/* Should not happen given nstandbys guard above, but be safe */
mylock_unlock(&r->lock);
@ -109,6 +154,17 @@ int repl_add_standby(replicator_t *r, repl_ep_t ep, const char *host, uint32_t p
}
standby_conn_t *s = &r->standbys[idx];
if (used_fallback_slot) {
/* bug-1474 review finding: this silently destroys the slot's
* WAL-retention guard below (via memset) -- log it loudly, mirroring
* repl_broadcast's existing "degraded but continuing" WARN
* convention (e.g. "standby %s:%u evicted", line ~314 above), so the
* discarded protection is operator-visible instead of silent. */
WARN("repl_add_standby: reusing guard-holding slot %d (host=%s:%u,"
" last_disconnected_lsn=%lu) -- no virgin slot available,"
" WAL-retention protection discarded",
idx, s->host, s->port, (unsigned long)s->last_disconnected_lsn);
}
memset(s, 0, sizeof(*s));
s->ep = ep;
s->port = port;
@ -719,6 +775,12 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
if (r->standbys[i].ep == ep) {
atomic_store_explicit(&r->standbys[i].in_catch_up, 0,
memory_order_release);
/* bug-1479: reset the eviction counter on re-entry to live
* broadcast -- otherwise a stale pending_ack_count carried over
* from before catch-up could immediately re-trip
* repl_broadcast's MAX_PENDING_ACKS eviction on the very next
* send, undoing the catch-up this function just completed. */
r->standbys[i].pending_ack_count = 0;
changed = 1;
break;
}
@ -738,6 +800,28 @@ void repl_complete_catchup(replicator_t *r, repl_ep_t ep)
pthread_mutex_unlock(&r->ack_lock);
}
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
uint64_t *join_applied_lsn_out)
{
if (r == NULL || join_applied_lsn_out == NULL)
return 0;
int found = 0;
mylock_lock(&r->lock);
for (int i = 0; i < MAX_STANDBYS; i++) {
if (r->standbys[i].ep == ep) {
if (atomic_load_explicit(&r->standbys[i].in_catch_up,
memory_order_acquire)) {
*join_applied_lsn_out = r->standbys[i].join_applied_lsn;
found = 1;
}
break;
}
}
mylock_unlock(&r->lock);
return found;
}
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn,
uint64_t oldest_wal_lsn)
{

219
src/controller/testsuite/configcompare.c Normal file
View file

@ -0,0 +1,219 @@
#define _GNU_SOURCE
#include "configcompare.h"
#include <errno.h>
#include <inttypes.h>
#include <openssl/sha.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
/* Bound on a member's config bytes this tool will read -- mirrors
* persist.c's PERSIST_CONFIG_BYTES_MAX_SIZE / config_repl.h's CONF_MAX_SIZE
* (1 MiB) exactly. Duplicated as its own local constant rather than
* including either header, for the same reason the resolve-then-read logic
* below is duplicated -- see read_member_config_sha256()'s doc comment. */
#define CONFIGCOMPARE_CONFIG_MAX_SIZE (1u * 1024u * 1024u)
/* Resolve `<conf_dir>/conf/current` -> `<conf_dir>/conf/<versioned file>`,
* read its raw bytes, and hash them into `out_sha`.
*
* DELIBERATE DUPLICATION -- this is the FOURTH independent copy of this
* exact resolve-then-read-then-hash logic in this codebase:
* 1. config_repl.c's conf_read_current_filename() + conf_read_raw()
* (the production implementation -- config replication).
* 2. config_repl.c's local_config_sha256() (activation-fence hash check;
* an inline copy for the same "no cross-file call needed here"
* reason as #4 below).
* 3. persist.c's read_active_config_bytes() (checkpoint config-bytes
* embedding; see its own doc comment for the full "why duplicate,
* not call" rationale -- avoiding a persist.c -> config_repl.c
* layering dependency).
* 4. This function. configcompare.c is compiled into the standalone
* xce_test CLI (meson.build's xce_test_src), which links ONLY
* libxce_dep / librtti_dep / struct2json_dep / readline_dep /
* history_dep / ncurses_dep / tinylog_dep / zlib_dep /
* c_capnproto_dep -- deliberately NOT config_repl.c, persist.c,
* cluster.c, or any other heavy controller object file. This is a
* lightweight, dependency-light offline CLI tool family
* (walcompare/snapdiff/snapverify all share the same constraint; no
* live RPC to any cluster member, consistent with every other tool
* in this family -- design.md "Regression Tooling"). Adding a
* config_repl.c dependency to link in one function would violate
* that boundary, so the ~25-line resolve+read is duplicated inline
* here instead, exactly as persist.c's read_active_config_bytes()
* already chose to do for the identical reason.
*
* Returns 0 with `out_sha` filled on success. Returns a negative errno on
* any failure to resolve or read: -ENOENT (no conf/current pointer, or it
* names a versioned file that is not present -- benign "not provisioned
* yet" as well as genuine absence, both surfaced identically to the
* caller as "unreadable"), -EINVAL (malformed pointer file: empty,
* embedded '/', wrong prefix, wrong hex length), -EIO (short
* read/zero-byte versioned file), -EFBIG (versioned file exceeds
* CONFIGCOMPARE_CONFIG_MAX_SIZE), or -ENOMEM. This is the caller's
* "unreadable/absent" signal -- distinct from and never conflated with a
* mismatched hash. */
static int read_member_config_sha256(const char* conf_dir, uint8_t out_sha[32])
{
if (!conf_dir || !*conf_dir)
return -EINVAL;
char head_path[4160];
snprintf(head_path, sizeof(head_path), "%s/conf/current", conf_dir);
FILE* hf = fopen(head_path, "r");
if (hf == NULL)
return -ENOENT; /* no config pushed to this member yet */
struct stat hst;
if (fstat(fileno(hf), &hst) == 0 && hst.st_size == 0) {
fclose(hf);
return -ENOENT; /* interrupted-write pointer file -- treat as absent */
}
char fname[256];
if (fgets(fname, (int)sizeof(fname), hf) == NULL) {
fclose(hf);
return -EIO;
}
fclose(hf);
fname[strcspn(fname, "\n")] = '\0';
if (strchr(fname, '/') != NULL)
return -EINVAL; /* corrupted pointer file -- embedded path separator */
if (strncmp(fname, "cluster.bin.", 12) != 0)
return -EINVAL;
const char* hex = fname + 12;
if (strlen(hex) != 12 || strspn(hex, "0123456789abcdef") != 12)
return -EINVAL;
char cfg_path[4416];
snprintf(cfg_path, sizeof(cfg_path), "%s/conf/%s", conf_dir, fname);
FILE* cf = fopen(cfg_path, "rb");
if (cf == NULL)
return -ENOENT; /* pointer references a file that is not there */
if (fseek(cf, 0, SEEK_END) != 0) { fclose(cf); return -EIO; }
long sz = ftell(cf);
if (sz < 0) { fclose(cf); return -EIO; }
if (fseek(cf, 0, SEEK_SET) != 0) { fclose(cf); return -EIO; }
if (sz == 0) {
fclose(cf);
return -EIO; /* zero-byte versioned config file: genuine corruption */
}
if ((size_t)sz > CONFIGCOMPARE_CONFIG_MAX_SIZE) {
fclose(cf);
return -EFBIG;
}
uint8_t* buf = (uint8_t*)malloc((size_t)sz);
if (buf == NULL) {
fclose(cf);
return -ENOMEM;
}
if (fread(buf, 1, (size_t)sz, cf) != (size_t)sz) {
free(buf);
fclose(cf);
return -EIO;
}
fclose(cf);
SHA256(buf, (size_t)sz, out_sha);
free(buf);
return 0;
}
static const char* sha256_hex(const uint8_t sha[32], char out[65])
{
static const char digits[] = "0123456789abcdef";
for (int i = 0; i < 32; i++) {
out[i * 2] = digits[(sha[i] >> 4) & 0xF];
out[i * 2 + 1] = digits[sha[i] & 0xF];
}
out[64] = '\0';
return out;
}
int configcompare_paths(const char* const* conf_paths, int n, FILE* out,
configcompare_report_t* rep)
{
configcompare_report_t local;
memset(&local, 0, sizeof(local));
local.first_divergent_index = -1;
if (!conf_paths || n < 2) {
if (rep)
*rep = local;
if (out)
fprintf(out,
"configcompare: nothing to compare (need >= 2 members, "
"got %d)\n", n);
return -EINVAL;
}
if (n > CONFIGCOMPARE_MAX_MEMBERS) {
if (rep)
*rep = local;
if (out)
fprintf(out, "configcompare: too many members (%d > max %d)\n", n,
CONFIGCOMPARE_MAX_MEMBERS);
return -EINVAL;
}
local.n = n;
int first_ok = -1;
char hexbuf[65];
for (int i = 0; i < n; i++) {
int rc = read_member_config_sha256(conf_paths[i], local.sha256[i]);
local.read_rc[i] = rc;
if (rc != 0) {
if (out)
fprintf(out,
"configcompare: member %d (%s): UNREADABLE: %s\n", i,
conf_paths[i] ? conf_paths[i] : "(null)",
strerror(-rc));
if (rep)
*rep = local;
/* Short-circuit on the first unreadable member, mirroring
* snapdiff_paths's early-return-on-open-failure style: a
* member we could not read is not evidence of matching OR
* mismatching configuration, so there is no point continuing
* to accumulate a verdict that will be discarded anyway. */
return rc;
}
if (out)
fprintf(out, "configcompare: member %d (%s): sha256=%s\n", i,
conf_paths[i], sha256_hex(local.sha256[i], hexbuf));
if (first_ok < 0) {
first_ok = i;
} else if (local.first_divergent_index < 0 &&
memcmp(local.sha256[i], local.sha256[first_ok], 32) != 0) {
local.first_divergent_index = i;
}
}
local.all_present = 1;
local.all_identical = (local.first_divergent_index < 0);
if (rep)
*rep = local;
if (!local.all_identical) {
if (out)
fprintf(out,
"configcompare: DIVERGENT -- member %d differs from "
"member %d\n",
local.first_divergent_index, first_ok);
return 1;
}
if (out)
fprintf(out, "configcompare: IDENTICAL across %d members\n", n);
return 0;
}

74
src/controller/testsuite/configcompare.h Normal file
View file

@ -0,0 +1,74 @@
#pragma once
/*
* configcompare.h — cross-member active-configuration comparator
* (config-upgrade-snapshot task 8.4, Requirement 4.2).
*
* Compares each cluster member's locally-held ACTIVE configuration content
* hash, recomputed FRESH from that member's own on-disk configuration file
* (never trusted from a cached/embedded tag), and reports whether every
* member agrees.
*
* A new sibling tool to walcompare, NOT an extension of it: walcompare's
* internal model is WAL-entry-indexed (lsn -> term, pcrc) -- an awkward fit
* for a single (tag, sha256) value per member with no LSN dimension
* (design.md "Regression Tooling"). Mirrors snapdiff_paths's exact
* signature/reporting shape instead.
*
* Offline/file-based: no live RPC to any cluster member, consistent with
* every other tool in this family (walcompare, snapdiff, snapverify).
*
* Strictly read-only: opens files O_RDONLY, mutates nothing.
*/
#include <stdint.h>
#include <stdio.h>
/* Bound on the number of members a single call can compare -- generous for
* any real cluster size this codebase targets; a caller with more members
* than this is almost certainly passing a malformed argument list. */
#define CONFIGCOMPARE_MAX_MEMBERS 32
typedef struct {
int n; /* member count passed in */
int read_rc[CONFIGCOMPARE_MAX_MEMBERS]; /* 0 = resolved+hashed ok,
* else -errno; entries at
* or past a short-circuit
* failure are unset (0)
* and must be ignored --
* see all_present */
uint8_t sha256[CONFIGCOMPARE_MAX_MEMBERS][32]; /* valid only where
* read_rc[i] == 0 */
int all_present; /* 1 iff every member (0..n-1) was resolved and
* hashed successfully */
int all_identical; /* 1 iff all_present && every member's sha256
* equals the first member's */
int first_divergent_index; /* -1 if all_identical or !all_present;
* else the first index (after the
* reference, member 0) whose hash
* differs from member 0's */
} configcompare_report_t;
/* Compare the active-configuration content hash of `n` members. Each
* `conf_paths[i]` is a member's `workdir/name` directory -- the SAME
* directory persist.c's `read_active_config_bytes()` and config_repl.c's
* `local_config_sha256()` both resolve `conf/current` (and the versioned
* config file it points at) under, matching where this spec's snapshot-embed
* step also reads from. NOT the raw `conf/current` file itself -- this
* function resolves `<path>/conf/current` and `<path>/conf/<versioned-file>`
* internally, exactly like those two functions do.
*
* Prints a human-readable report to `out` (may be NULL) and fills `*rep`
* (may be NULL). Returns:
* 0 if every member's config was readable and all hashes are identical
* 1 if every member's config was readable but at least one diverges
* (see `*rep->first_divergent_index` for which)
* <0 (negative errno) if `n < 2` (nothing to compare), `n` exceeds
* CONFIGCOMPARE_MAX_MEMBERS, or any member's config could not be
* resolved/read -- mirrors snapdiff_paths's early-return-on-open-
* failure style: the first unreadable member short-circuits the scan,
* and `*rep` is still filled with whatever was learned about the
* members checked before the failure, so the report can say which
* member and why. A member that could not be read is NOT evidence of
* matching OR mismatching configuration -- it is a distinct third
* outcome from "identical" and "divergent". */
int configcompare_paths(const char* const* conf_paths, int n, FILE* out,
configcompare_report_t* rep);

8
src/controller/testsuite/controller_test.c
View file

@ -69,6 +69,7 @@ extern void test_waldump(void);
extern void test_walcompare(void);
extern void test_snapverify(void);
extern void test_snapdiff(void);
extern void test_configcompare(void);
extern void test_drive_async_config(void);
extern void test_slowlane(void);
extern void test_commit_admission_pool(void);
@ -88,6 +89,9 @@ extern void test_apply_pump_integration(void);
extern void test_job_array_apply(void);
extern void test_job_array_fence(void);
extern void test_job_array_replay_fence(void);
extern void test_upgrade_snapshot_restore(void);
extern void test_upgrade_snapshot_apply(void);
extern void test_bug857_regression(void);
void setUp(void) { return; }
@ -226,6 +230,7 @@ int main(int argc, char* argv[])
TEST(test_walcompare);
TEST(test_snapverify);
TEST(test_snapdiff);
TEST(test_configcompare);
TEST(test_drive_async_config);
TEST(test_slowlane);
TEST(test_commit_admission_pool);
@ -245,6 +250,9 @@ int main(int argc, char* argv[])
TEST(test_job_array_apply);
TEST(test_job_array_fence);
TEST(test_job_array_replay_fence);
TEST(test_upgrade_snapshot_restore);
TEST(test_upgrade_snapshot_apply);
TEST(test_bug857_regression);
return UNITY_END();
}

179
src/controller/testsuite/snapverify.c
View file

@ -9,8 +9,12 @@
#include <unistd.h>
/* Mirrors persist.c's SNAPSHOT_FORMAT_VERSION / header layout exactly — see
* persist.c write_snapshot() (~line 1758) and load_snapshot() (~line 2325). */
#define SNAP_FORMAT_VERSION 1u
* persist.c write_snapshot() (~line 1758) and load_snapshot() (~line 2325).
* config-upgrade-snapshot task 1 bumped persist.c's SNAPSHOT_FORMAT_VERSION
* 1->2 (host/queue admin-state frames migrated to capnp, Req 1.9/5.7) --
* this MUST track that bump, or this tool hard-rejects every real
* checkpoint the current codebase writes (task 8.4 finding). */
#define SNAP_FORMAT_VERSION 2u
#define SNAP_HEADER_MAX (16 + 1 + 8 + 32) /* magic+lsn + optional cfgref tail */
static void fail(snapverify_report_t* rep, int rc, off_t off, const char* why)
@ -78,8 +82,17 @@ int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep)
}
memcpy(&local.checkpoint_lsn, header + 8, 8);
if (n >= SNAP_HEADER_MAX && header[16] != 0)
local.has_cfgref = 1;
if (n >= SNAP_HEADER_MAX) {
/* Decode the tag/hash UNCONDITIONALLY whenever the header extends
* this far, regardless of the presence byte -- so a caller can
* inspect the raw bytes even in edge cases. has_cfgref keeps its
* existing meaning (whether those bytes are semantically
* meaningful) untouched. */
memcpy(&local.cfgref_tag, header + 17, 8);
memcpy(local.cfgref_sha, header + 25, 32);
if (header[16] != 0)
local.has_cfgref = 1;
}
off_t consumed = (off_t)n;
@ -200,11 +213,11 @@ int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep)
local.queue_count = queue_count;
for (uint32_t i = 0; i < queue_count; i++) {
off_t len_off = consumed;
uint32_t name_len = 0;
if (read(fd, &name_len, sizeof(name_len)) != (ssize_t)sizeof(name_len)) {
off_t len_off = consumed;
uint32_t plen = 0;
if (read(fd, &plen, sizeof(plen)) != (ssize_t)sizeof(plen)) {
close(fd);
fail(&local, -EIO, len_off, "short read on queue name_len");
fail(&local, -EIO, len_off, "short read on queue payload_len");
if (rep)
*rep = local;
if (out)
@ -212,13 +225,13 @@ int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep)
path, i, local.fail_reason, (intmax_t)len_off);
return -EIO;
}
consumed += (off_t)sizeof(name_len);
consumed += (off_t)sizeof(plen);
if (name_len == 0 || (off_t)name_len > st.st_size - consumed) {
if (plen == 0 || (off_t)plen > st.st_size - consumed) {
close(fd);
char why[160];
snprintf(why, sizeof(why),
"bad name_len=%u (remaining=%jd)", name_len,
"bad payload_len=%u (remaining=%jd)", plen,
(intmax_t)(st.st_size - consumed));
fail(&local, -EIO, len_off, why);
if (rep)
@ -230,9 +243,9 @@ int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep)
return -EIO;
}
if (lseek(fd, name_len, SEEK_CUR) < 0) {
if (lseek(fd, plen, SEEK_CUR) < 0) {
close(fd);
fail(&local, -EIO, consumed, "seek past queue name failed");
fail(&local, -EIO, consumed, "seek past queue payload failed");
if (rep)
*rep = local;
if (out)
@ -240,21 +253,138 @@ int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep)
local.fail_reason);
return -EIO;
}
consumed += (off_t)name_len;
consumed += (off_t)plen;
local.queues_ok++;
}
uint32_t status = 0;
if (read(fd, &status, sizeof(status)) != (ssize_t)sizeof(status)) {
/* Host section (config-upgrade-snapshot task 4/8.4, SNAPSHOT_FORMAT_
* VERSION=2): new after the queue section (design.md "Checkpoint File
* Layout", persist.c load_snapshot_entities()/write_snapshot()). Same
* [payload_len][bytes] per-frame structural shape as the job/queue
* loops above -- this tool validates structure only, never decodes the
* capnp payload. */
uint32_t host_count = 0;
if (read(fd, &host_count, sizeof(host_count)) !=
(ssize_t)sizeof(host_count)) {
close(fd);
fail(&local, -EIO, consumed, "short read on host_count");
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: %s (offset %jd)\n", path,
local.fail_reason, (intmax_t)consumed);
return -EIO;
}
consumed += (off_t)sizeof(host_count);
local.host_count = host_count;
for (uint32_t i = 0; i < host_count; i++) {
off_t len_off = consumed;
uint32_t plen = 0;
if (read(fd, &plen, sizeof(plen)) != (ssize_t)sizeof(plen)) {
close(fd);
fail(&local, -EIO, consumed, "short read on queue status");
fail(&local, -EIO, len_off, "short read on host payload_len");
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: queue %u: %s (offset %jd)\n",
path, i, local.fail_reason, (intmax_t)consumed);
fprintf(out, "snapverify: %s: host %u: %s (offset %jd)\n",
path, i, local.fail_reason, (intmax_t)len_off);
return -EIO;
}
consumed += (off_t)sizeof(status);
local.queues_ok++;
consumed += (off_t)sizeof(plen);
if (plen == 0 || (off_t)plen > st.st_size - consumed) {
close(fd);
char why[160];
snprintf(why, sizeof(why),
"bad payload_len=%u (remaining=%jd)", plen,
(intmax_t)(st.st_size - consumed));
fail(&local, -EIO, len_off, why);
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: host %u: %s (length field at "
"offset %jd)\n",
path, i, local.fail_reason, (intmax_t)len_off);
return -EIO;
}
if (lseek(fd, plen, SEEK_CUR) < 0) {
close(fd);
fail(&local, -EIO, consumed, "seek past host payload failed");
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: host %u: %s\n", path, i,
local.fail_reason);
return -EIO;
}
consumed += (off_t)plen;
local.hosts_ok++;
}
/* Config-bytes section (config-upgrade-snapshot task 2.3/8.4): a
* single flat length-prefixed blob appended LAST, after the host
* section -- NOT a [count][frame...] shape like job/queue/host above,
* since there is exactly one active configuration, never a set
* (persist.c write_snapshot()). The length field itself may be
* legitimately ABSENT (a file that ends right after the host section
* is still a valid, if slightly older, fmt=2-consistent shape) --
* distinguish that from the length field being present-but-zero
* (persist.c explicitly writes an empty-but-present section when no
* local config bytes exist yet) and from a genuinely truncated length
* field (a short, non-zero partial read -- real corruption). */
uint32_t cfg_bytes_len = 0;
ssize_t cfg_len_n = read(fd, &cfg_bytes_len, sizeof(cfg_bytes_len));
if (cfg_len_n == 0) {
/* Clean EOF right after the host section -- no config-bytes
* section present at all. Not a defect. */
local.has_config_bytes_section = 0;
} else if (cfg_len_n != (ssize_t)sizeof(cfg_bytes_len)) {
close(fd);
fail(&local, -EIO, consumed, "short read on config_bytes_len");
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: %s (offset %jd)\n", path,
local.fail_reason, (intmax_t)consumed);
return -EIO;
} else {
off_t len_off = consumed;
consumed += (off_t)sizeof(cfg_bytes_len);
local.has_config_bytes_section = 1;
local.config_bytes_len = cfg_bytes_len;
if ((off_t)cfg_bytes_len > st.st_size - consumed) {
close(fd);
char why[160];
snprintf(why, sizeof(why),
"bad config_bytes_len=%u (remaining=%jd)", cfg_bytes_len,
(intmax_t)(st.st_size - consumed));
fail(&local, -EIO, len_off, why);
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: %s (length field at offset "
"%jd)\n",
path, local.fail_reason, (intmax_t)len_off);
return -EIO;
}
if (cfg_bytes_len > 0) {
if (lseek(fd, cfg_bytes_len, SEEK_CUR) < 0) {
close(fd);
fail(&local, -EIO, consumed,
"seek past config bytes failed");
if (rep)
*rep = local;
if (out)
fprintf(out, "snapverify: %s: %s\n", path,
local.fail_reason);
return -EIO;
}
consumed += (off_t)cfg_bytes_len;
}
}
close(fd);
@ -277,8 +407,11 @@ int snapverify_path(const char* path, FILE* out, snapverify_report_t* rep)
if (out)
fprintf(out,
"snapverify: %s: OK (lsn=%" PRIu64 " fmt=%u jobs=%u queues=%u "
"size=%jd)\n",
"hosts=%u cfgref_tag=%016" PRIx64 " cfg_bytes_len=%u "
"has_cfg_bytes=%d size=%jd)\n",
path, local.checkpoint_lsn, local.format_version,
local.job_count, local.queue_count, (intmax_t)local.file_size);
local.job_count, local.queue_count, local.host_count,
local.cfgref_tag, local.config_bytes_len,
local.has_config_bytes_section, (intmax_t)local.file_size);
return 0;
}

44
src/controller/testsuite/snapverify.h
View file

@ -1,14 +1,20 @@
#pragma once
/*
* snapverify.h — read-only checkpoint-file structural inspector/validator
* (bug-1389 triage: OPCode_snapshotTransfer chunk-content corruption).
* (bug-1389 triage: OPCode_snapshotTransfer chunk-content corruption;
* extended by config-upgrade-snapshot task 8.4, Requirement 4.2, for the
* SNAPSHOT_FORMAT_VERSION=2 host + config-bytes sections and to decode --
* not merely report the presence of -- the header's cfgref tag/hash).
*
* Parses the on-disk checkpoint layout persist.c's load_snapshot() /
* load_snapshot_entities() use (design.md "Checkpoint File Layout") WITHOUT
* restoring any job/queue into a live cluster_t — a pure structural walk, so
* it needs no jobmgr/qmgr context and is safe to run standalone against a
* snap-*.capnp (locally generated) or snap-*.bin (received via
* ha_handle_snapshot_chunk, ha.c) file on disk.
* restoring any job/queue/host into a live cluster_t — a pure structural
* walk, so it needs no jobmgr/qmgr/workermgr context and is safe to run
* standalone against a snap-*.capnp (locally generated) or snap-*.bin
* (received via ha_handle_snapshot_chunk, ha.c) file on disk. This tool
* validates STRUCTURE only, never content: job/queue/host frame payloads
* and the config-bytes section are walked past via their length prefix,
* never capnp-decoded or interpreted.
*
* Strictly read-only: opens O_RDONLY, mutates nothing.
*/
@ -18,7 +24,17 @@
typedef struct {
uint64_t checkpoint_lsn;
int has_cfgref; /* config-ref tail present in the header */
int has_cfgref; /* config-ref tail present in the header
* (header[16] != 0) -- semantic presence,
* distinct from cfgref_tag/cfgref_sha below
* being decoded */
uint64_t cfgref_tag; /* header cfgref tag, decoded whenever the
* header extends to SNAP_HEADER_MAX bytes,
* REGARDLESS of has_cfgref -- meaningful
* only when has_cfgref is also 1 */
uint8_t cfgref_sha[32]; /* header cfgref sha256, same decode-
* unconditionally-but-gate-meaning-on-
* has_cfgref discipline as cfgref_tag */
int has_entity_section; /* fmt + job_count present past the header;
* 0 = legacy position-only checkpoint */
uint32_t format_version;
@ -27,6 +43,22 @@ typedef struct {
uint32_t queue_count; /* as declared in the file (0 if legacy, or if the
* job-frame walk failed before reaching it) */
uint32_t queues_ok; /* queue frames structurally walked before any failure */
uint32_t host_count; /* as declared in the file (0 if legacy, or if an
* earlier section's walk failed before reaching
* it) */
uint32_t hosts_ok; /* host frames structurally walked before any failure */
int has_config_bytes_section; /* 1 iff the trailing config-bytes
* section's 4-byte length field
* itself was present and readable
* (i.e. the file extends past the
* host section at all) -- TRUE even
* when that length decodes to 0, and
* FALSE (not a failure) when the
* file simply ends right after the
* host section */
uint32_t config_bytes_len; /* decoded length of the config-bytes
* section; valid only when
* has_config_bytes_section is 1 */
off_t file_size;
off_t consumed; /* bytes structurally accounted for */
int ok; /* 1 = well-formed, consumed == file_size */

174
src/controller/testsuite/test.c
View file

@ -15,6 +15,7 @@
#include <cjson/cJSON.h>
#include <cjson/s2j.h>
#include <openssl/sha.h> /* SHA256() — cmd_upgradeconfig hashes the config blob */
#include "client.h"
#include "config.h" /* XCE_MAXNAME_LEN */
@ -27,6 +28,7 @@
#include "waldump.h" /* offline WAL inspector (cmd_waldump) */
#include "snapdiff.h" /* byte-level checkpoint-file comparator (cmd_snapdiff) */
#include "snapverify.h" /* offline checkpoint-file structural validator (cmd_snapverify) */
#include "configcompare.h" /* cross-member active-config comparator (cmd_configcompare) */
#include "xce/cluster.h" /* xce_parse_peers, xce_cluster_config_t */
#include "xce/types.h"
@ -121,6 +123,8 @@ extern int cmd_resval(int, char*[]);
extern int cmd_jobkill(int, char*[]);
extern int cmd_jobckpt(int, char*[]);
extern int cmd_qctrl(int, char*[]);
extern int cmd_hostctrl(int, char*[]);
extern int cmd_upgradeconfig(int, char*[]);
extern int cmd_runtasks(int, char*[]);
extern int cmd_setpend(int, char*[]);
extern int cmd_jobinfo(int, char*[]);
@ -133,6 +137,7 @@ extern int cmd_waldump(int, char*[]);
extern int cmd_walcompare(int, char*[]);
extern int cmd_snapverify(int, char*[]);
extern int cmd_snapdiff(int, char*[]);
extern int cmd_configcompare(int, char*[]);
static int worker_house_keeping(void*);
/* Session verdict / loop state (xce-test-e2e-primitives task 1.1).
@ -194,6 +199,7 @@ static command_t top_commands[] = {
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes" },
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure" },
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files" },
{ "configcompare", cmd_configcompare, "Compare active config hash across members" },
{ "quit", cmd_quit, "Quit xce test" },
{ (char*)NULL, (cmd_func_t)NULL, (char*)NULL }
};
@ -227,6 +233,9 @@ static command_t client_commands[] = {
{ "reslist", cmd_reslist, "Display cluster resources" },
{ "workerlist", cmd_workerlist, "list workers" },
{ "qctrl", cmd_qctrl, "queue control" },
{ "hostctrl", cmd_hostctrl, "host control" },
{ "upgradeconfig", cmd_upgradeconfig,
"submit a config upgrade (cluster.bin file)" },
{ "joblist", cmd_joblist, "list jobs" },
{ "expect-jobs", cmd_expect_jobs, "assert current job count" },
{ "expect-job", cmd_expect_job, "assert a JobId is present" },
@ -237,6 +246,7 @@ static command_t client_commands[] = {
{ "walcompare", cmd_walcompare, "Diff member committed WAL prefixes" },
{ "snapverify", cmd_snapverify, "Verify checkpoint file structure" },
{ "snapdiff", cmd_snapdiff, "Byte-diff two checkpoint files" },
{ "configcompare", cmd_configcompare, "Compare active config hash across members" },
{ "shutdown", cmd_shutdown, "shutdown xce controller" },
{ "jobrun", cmd_jobrun, "run a job" },
{ "jobsub", cmd_jobsub, "submit a job" },
@ -856,6 +866,27 @@ int cmd_snapdiff(int argc, char* argv[])
return (rc == 0) ? 0 : -1;
}
/* configcompare <member_dir> <member_dir> [more...] — cross-member
* active-configuration comparator (config-upgrade-snapshot task 8.4, Req
* 4.2). Each path is a member's `workdir/name` conf-root directory (the
* same directory persist.c's read_active_config_bytes() and config_repl.c's
* local_config_sha256() resolve `conf/current` under) — NOT a raw config
* file path. Recomputes each member's active-config sha256 fresh from its
* own on-disk bytes and reports whether all members match. Returns 0 if
* all members are present and identical, nonzero if any member is
* unreadable or divergent. */
int cmd_configcompare(int argc, char* argv[])
{
if (argc < 3) {
fprintf(stderr, "usage: configcompare <member_dir> <member_dir> [more...]\n");
return -1;
}
configcompare_report_t rep;
int rc = configcompare_paths((const char* const*)&argv[1], argc - 1, stdout, &rep);
return (rc == 0) ? 0 : -1;
}
int cmd_up(int argc, char* argv[])
{
if (active_ctx->parent != NULL) {
@ -995,7 +1026,15 @@ int cmd_reslist(int argc, char* argv[])
static char* worker_status_str(uint32_t status)
{
if ((status & XCE_WORKER_UNAVAIL) != 0) {
/* Admin-set CLOSED takes priority over the dynamic UNAVAIL bit: the
* config-upgrade-snapshot soak (task 8.6) needs to assert a host's
* administratively-set status specifically, and a host closed via
* hostctrl that later disconnects (e.g. its xce_test `worker`/`join`
* connection exits after the script ends) would otherwise print only
* "UNAVAIL", masking the admin bit this check exists to verify. */
if ((status & XCE_WORKER_CLOSED) != 0) {
return "CLOSED";
} else if ((status & XCE_WORKER_UNAVAIL) != 0) {
return "UNAVAIL";
} else if ((status & XCE_WORKER_LOCKEDU) != 0) {
return "LOCKU";
@ -2199,6 +2238,139 @@ int cmd_qctrl(int argc, char* argv[])
return 0;
}
/* hostctrl -o|-c -h <hostname> — administratively open/close a host
* (xce_hostctrl_t{action, hostname} via xcecli_hostctrl). Mirrors cmd_qctrl's
* exact getopt shape; only -o/-c are wired (matching xce_hostctrl_req's own
* accepted action range, request.c — HOST_TAG/HOST_UNTAG exist in the enum
* but are rejected -EINVAL server-side today, so no flag is offered for them
* here). Used by the config-upgrade chaotic soak (task 8.6) to set a host's
* admin status before/around a config upgrade and assert it survives. */
int cmd_hostctrl(int argc, char* argv[])
{
int opt;
int rc;
xce_hostctrl_t hostctrl;
int action = HOST_DO_NONE;
char* hostname = NULL;
optind = 1;
while ((opt = getopt(argc, argv, "och:")) != -1) {
switch (opt) {
case 'o':
action = HOST_OPEN;
break;
case 'c':
action = HOST_CLOSE;
break;
case 'h':
hostname = strdup(optarg);
break;
default:
break;
}
}
if (hostname == NULL) {
fprintf(stderr, "you need to specify host name with -h\n");
return -1;
}
if (action == HOST_DO_NONE) {
fprintf(stderr, "you need to specify -o (open) or -c (close)\n");
free(hostname);
return -1;
}
hostctrl.action = action;
hostctrl.hostname = hostname;
rc = xcecli_hostctrl(active_ctx->ctx, &hostctrl);
if (rc < 0) {
perror("xcecli_hostctrl");
free(hostname);
return rc;
}
printf("Operation on host %s done\n", hostname);
free(hostname);
return 0;
}
/* upgradeconfig <cluster.bin> — submit a configuration upgrade
* (config-upgrade-snapshot task 8.6). Reads the given capnp-encoded
* ClusterConfig blob (the same format `xceadmin config build` produces and
* deploy-ha.sh installs as cluster.bin), computes its SHA-256 (OpenSSL
* SHA256(), matching config_repl.c's do_upgrade_config_with_bytes transit
* verification), and drives it through xcecli_upgradeconfig ->
* xce_upgradeconfig_req_raw (OP_upgradeConfig). Mirrors cmd_snapdiff's
* bounds-checked file-reading style. */
int cmd_upgradeconfig(int argc, char* argv[])
{
if (argc != 2) {
fprintf(stderr, "usage: upgradeconfig <cluster.bin>\n");
return -1;
}
const char* path = argv[1];
FILE* fp = fopen(path, "rb");
if (fp == NULL) {
fprintf(stderr, "upgradeconfig: cannot open %s: %s\n", path,
strerror(errno));
return -1;
}
if (fseek(fp, 0, SEEK_END) != 0) {
fprintf(stderr, "upgradeconfig: fseek failed on %s: %s\n", path,
strerror(errno));
fclose(fp);
return -1;
}
long fsz = ftell(fp);
if (fsz <= 0) {
fprintf(stderr, "upgradeconfig: %s is empty or ftell failed\n", path);
fclose(fp);
return -1;
}
rewind(fp);
uint8_t* bytes = (uint8_t*)malloc((size_t)fsz);
if (bytes == NULL) {
fprintf(stderr, "upgradeconfig: out of memory reading %s\n", path);
fclose(fp);
return -1;
}
size_t nread = fread(bytes, 1, (size_t)fsz, fp);
fclose(fp);
if (nread != (size_t)fsz) {
fprintf(stderr, "upgradeconfig: short read on %s (%zu of %ld bytes)\n",
path, nread, fsz);
free(bytes);
return -1;
}
uint8_t sha256[SHA256_DIGEST_LENGTH];
SHA256(bytes, (size_t)fsz, sha256);
int rc = xcecli_upgradeconfig(active_ctx->ctx, sha256,
sizeof(sha256), bytes, (size_t)fsz);
free(bytes);
if (rc < 0) {
fprintf(stderr, "upgradeconfig: failed (rc=%d): %s\n", rc,
strerror(-rc));
return rc;
}
printf("Config upgrade submitted from %s (%ld bytes)\n", path, fsz);
return 0;
}
int cmd_runtasks(int argc, char* argv[])
{
xce_taskhost_t hosts[MAX_TASK_HOSTS];

199
src/controller/testsuite/test_apply_committed_entry.c
View file

@ -496,6 +496,202 @@ static void test_ace_worker_ctrl_join_restores_and_notifies(void)
printf("test_ace_worker_ctrl_join_restores_and_notifies: PASS\n");
}
/* Builds a [uint32 detail][capnp Worker] WAL_WORKER_CTRL payload for a
* worker named `name` carrying `status`. Mirrors the JOIN test's inline
* construction above, factored out so the UPDATE-case tests below (bug-1472)
* can build a payload against an arbitrary pre-existing worker name/status.
* Caller frees *pkt_out. */
static void tace_build_worker_ctrl_pkt(const char* name, uint32_t status,
uint32_t detail, uint8_t** pkt_out,
size_t* total_out)
{
hostattrs_t attrs;
memset(&attrs, 0, sizeof(attrs));
attrs.hosttype = (char*)"unknown";
attrs.hostmodel = (char*)"unknown";
xce_worker_t wdef;
memset(&wdef, 0, sizeof(wdef));
wdef.name = (char*)name;
wdef.attrs = &attrs;
wdef.status = status;
uint8_t* wbuf = NULL;
size_t wsize = 0;
TEST_ASSERT_EQUAL_INT(0, tace_encode_worker(&wdef, &wbuf, &wsize));
size_t total = sizeof(detail) + wsize;
uint8_t* pkt = malloc(total);
TEST_ASSERT_NOT_NULL(pkt);
memcpy(pkt, &detail, sizeof(detail));
memcpy(pkt + sizeof(detail), wbuf, wsize);
free(wbuf);
*pkt_out = pkt;
*total_out = total;
}
/* bug-1472: worker_ctrl() (HOST_OPEN/HOST_CLOSE, worker.c:732-794) mutates
* only the admin-mutable status bits on the member that received the RPC,
* then fires the SAME WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_UPDATE apply path
* on every OTHER member (standby, or this member's own WAL replay). Before
* the fix, worker_from_capnp's existing-worker guard returned -EEXIST
* unconditionally, silently discarding the decoded status change -- an
* admin hostctrl -c/-o against an already-joined host never propagated.
* This proves an UPDATE entry against an existing worker actually mutates
* it and fires the notification, matching worker_ctrl's own encode (which
* always carries the worker's post-mutation w->host.status). */
static void test_ace_worker_ctrl_update_applies_to_existing_worker(void)
{
tace_setup();
/* Pre-existing worker (e.g. from an earlier JOIN), currently open. */
hostattrs_t attrs;
memset(&attrs, 0, sizeof(attrs));
attrs.hosttype = (char*)"unknown";
attrs.hostmodel = (char*)"unknown";
xce_worker_t wdef;
memset(&wdef, 0, sizeof(wdef));
wdef.name = (char*)"w-update";
wdef.attrs = &attrs;
wdef.status = 0;
TEST_ASSERT_NOT_NULL(makeWorker(g_tace_c.workermgr, &wdef));
uint8_t* pkt = NULL;
size_t total = 0;
tace_build_worker_ctrl_pkt("w-update",
XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN,
(uint32_t)HOOK_DETAIL_WORKER_UPDATE, &pkt,
&total);
xce_add_hook(&g_tace_c.hooksys, XCE_CONTROLLER_HOOK_WORKER, tace_spy_hook,
NULL);
int rc = apply_committed_entry(&g_tace_p, WAL_WORKER_CTRL, 41, pkt,
(uint32_t)total);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
"an UPDATE against an already-existing worker must succeed (0), not "
"-EEXIST -- bug-1472: -EEXIST silently discarded the status change");
worker_t* w = NULL;
lock(g_tace_c.workermgr, RTTI_LOCK_READ);
key_find(g_tace_c.workermgr, (void*)"w-update", (void**)&w);
unlock(g_tace_c.workermgr);
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_TRUE_MESSAGE(
(w->host.status & (XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN))
== (XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN),
"the existing worker's admin status must be replaced with the "
"decoded UPDATE payload's, not silently discarded");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_tace_hook_fires,
"a genuinely-applied UPDATE must fire exactly one notification");
free(pkt);
tace_teardown();
printf("test_ace_worker_ctrl_update_applies_to_existing_worker: PASS\n");
}
/* worker_ctrl's HOST_OPEN clears the closed bits via assignment-style
* &= ~(...) (worker.c:773), not a pure OR -- so the fix must REPLACE the
* existing worker's admin-masked bits with the decoded payload's, not OR
* them in (an OR-based fix would leave a re-opened worker stuck closed
* forever). This guards that exact semantics: apply an UPDATE encoding
* status=0 (no admin bits, i.e. a HOST_OPEN) against a worker that starts
* admin-closed and confirms the closed bits are actually cleared. */
static void test_ace_worker_ctrl_update_replaces_not_ors_admin_bits(void)
{
tace_setup();
hostattrs_t attrs;
memset(&attrs, 0, sizeof(attrs));
attrs.hosttype = (char*)"unknown";
attrs.hostmodel = (char*)"unknown";
xce_worker_t wdef;
memset(&wdef, 0, sizeof(wdef));
wdef.name = (char*)"w-reopen";
wdef.attrs = &attrs;
wdef.status = XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN;
TEST_ASSERT_NOT_NULL(makeWorker(g_tace_c.workermgr, &wdef));
uint8_t* pkt = NULL;
size_t total = 0;
tace_build_worker_ctrl_pkt("w-reopen", 0,
(uint32_t)HOOK_DETAIL_WORKER_UPDATE, &pkt,
&total);
int rc = apply_committed_entry(&g_tace_p, WAL_WORKER_CTRL, 42, pkt,
(uint32_t)total);
TEST_ASSERT_EQUAL_INT(0, rc);
worker_t* w = NULL;
lock(g_tace_c.workermgr, RTTI_LOCK_READ);
key_find(g_tace_c.workermgr, (void*)"w-reopen", (void**)&w);
unlock(g_tace_c.workermgr);
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
w->host.status & (XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN),
"a HOST_OPEN UPDATE must CLEAR the closed bits (replace semantics) "
"-- an OR-based fix would incorrectly leave the worker closed");
free(pkt);
tace_teardown();
printf("test_ace_worker_ctrl_update_replaces_not_ors_admin_bits: PASS\n");
}
/* Regression guard (bug-1472 fix must NOT change JOIN's behavior): a
* duplicate JOIN replay against an already-registered worker (e.g. WAL
* replay / resync redelivery) must remain a genuine no-op -- -EEXIST
* translated to 0 by apply_committed_entry, no status mutation, no
* notification. */
static void test_ace_worker_ctrl_join_duplicate_existing_is_noop(void)
{
tace_setup();
hostattrs_t attrs;
memset(&attrs, 0, sizeof(attrs));
attrs.hosttype = (char*)"unknown";
attrs.hostmodel = (char*)"unknown";
xce_worker_t wdef;
memset(&wdef, 0, sizeof(wdef));
wdef.name = (char*)"w-dupjoin";
wdef.attrs = &attrs;
wdef.status = 0;
TEST_ASSERT_NOT_NULL(makeWorker(g_tace_c.workermgr, &wdef));
uint8_t* pkt = NULL;
size_t total = 0;
tace_build_worker_ctrl_pkt("w-dupjoin",
XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN,
(uint32_t)HOOK_DETAIL_WORKER_JOIN, &pkt,
&total);
xce_add_hook(&g_tace_c.hooksys, XCE_CONTROLLER_HOOK_WORKER, tace_spy_hook,
NULL);
int rc = apply_committed_entry(&g_tace_p, WAL_WORKER_CTRL, 43, pkt,
(uint32_t)total);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
"apply_committed_entry translates worker_from_capnp's -EEXIST to 0 "
"(idempotent no-op)");
worker_t* w = NULL;
lock(g_tace_c.workermgr, RTTI_LOCK_READ);
key_find(g_tace_c.workermgr, (void*)"w-dupjoin", (void**)&w);
unlock(g_tace_c.workermgr);
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
w->host.status & (XCE_WORKER_CLOSED | XCE_WORKER_CLOSED_BY_ADMIN),
"a duplicate JOIN replay must NOT mutate the existing worker's "
"status -- this pre-existing JOIN behavior is intentionally "
"unchanged by the bug-1472 UPDATE fix");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_tace_hook_fires,
"a duplicate JOIN no-op must not fire a notification");
free(pkt);
tace_teardown();
printf("test_ace_worker_ctrl_join_duplicate_existing_is_noop: PASS\n");
}
/* ── WAL_JOB_ARRAY_SUBMIT (job-array-single-wal-record task 3.2) ────────── */
static job_submit_data_t tace_array_make_spec(void)
@ -748,6 +944,9 @@ void test_apply_committed_entry(void)
RUN_TEST(test_ace_job_state_transition_rejects_backward_suppresses_notify);
RUN_TEST(test_ace_queue_ctrl_updates_and_notifies);
RUN_TEST(test_ace_worker_ctrl_join_restores_and_notifies);
RUN_TEST(test_ace_worker_ctrl_update_applies_to_existing_worker);
RUN_TEST(test_ace_worker_ctrl_update_replaces_not_ors_admin_bits);
RUN_TEST(test_ace_worker_ctrl_join_duplicate_existing_is_noop);
RUN_TEST(test_ace_job_array_submit_constructs_all_and_notifies_once);
RUN_TEST(test_ace_job_array_submit_idempotent_replay_no_extra_notify);
#ifdef XCE_TESTING

513
src/controller/testsuite/test_bug857_regression.c Normal file
View file

@ -0,0 +1,513 @@
/*
* test_bug857_regression.c — config-upgrade-snapshot task 8.5
*
* Reproduces the bug-857 materialized-state divergence on the PRE-fix
* mechanism and proves it converges on the CURRENT (post-task-5) one, both
* at the controller_unit_test level, entirely within this one file.
*
* ── Investigation summary (why this file looks the way it does) ────────
*
* Task 8.5's original brief hypothesized bug-857's divergence as "a job
* whose queue was removed by a config upgrade replays into its original
* queue on one member and gets re-homed to the synthetic `lost_and_found`
* queue on another, depending on each member's own restart timing."
* Investigation (this task) found that mechanism does not exist as
* implemented code, on ANY commit: `create_lost_and_found` (config_repl.c)
* only ever creates the sentinel queue/host objects — no code path in
* job.c/queue.c/worker.c ever reassigns an EXISTING job's queue reference
* onto them. `restore_job`/`restore_job_array` (job.c) never consult
* `c->qmgr` at all; `restore_queue_state` (queue.c) explicitly SKIPS
* (returns -ENOENT) rather than re-homes when the target queue is absent.
* `create_lost_and_found`'s body is also unconditionally `#ifndef
* XCE_TESTING`'d out. bug-857's buglog entry itself already flagged this
* as "NOT empirically reproduced yet" — the re-homing detail was TLA+-
* derived aspiration, not observed behavior.
*
* design.md's Requirements Traceability table (row 6.2) originally
* assigned this requirement to "HA soak new scenario" — written under the
* same since-corrected hypothesis. The real, code-backed divergence this
* file reproduces instead is CHECKPOINT-CONTENT divergence: pre-task-5,
* `apply_committed_entry`'s WAL_CONFIG_MARKER case was a no-op (the
* generic `default:` case) and a checkpoint's LSN/content was determined
* SOLELY by `schedule_checkpoint`'s independent per-member WAL-record-count
* threshold (persist.c) — completely decoupled from the config-change
* marker's own position in the shared committed log. Two members reaching
* that threshold at different real-world moments (because other activity —
* more job submissions — is interleaved around the upgrade) materialize
* genuinely different job/queue content, with NEITHER checkpoint anchored
* to the marker at all: exactly "no snapshot pinning an identical
* post-upgrade state" (bug-857's own root-cause wording). Post-task-5,
* `apply_committed_entry`'s WAL_CONFIG_MARKER case forces
* `persist_checkpoint_marker_triggered()` synchronously, INLINE, the
* instant the marker entry itself is dispatched — before the per-LSN
* apply loop ever reaches any later entry — so every member's
* marker-anchored checkpoint captures identical content regardless of its
* own processing pace.
*
* Both scenarios below run entirely within controller_unit_test — no
* cluster.c linkage (cluster_init/cluster_build_resources), no scratch
* worktree, no live multi-process cluster needed. See this task's Status
* Report / design.md's 6.2 traceability-row correction note for the full
* decision trail.
*
* ── Scenario 1: test_bug857_threshold_checkpoint_diverges_across_members ──
* Reproduces the PRE-task-5 mechanism for real: two members receive the
* identical physical WAL (job1, config-change marker, job2), but their
* entity state and checkpoint are driven manually via restore_job() +
* persist_checkpoint() — the SAME production functions
* apply_committed_entry()/schedule_checkpoint() would themselves call —
* WITHOUT ever routing the marker's LSN through apply_committed_entry()
* itself (which, on the CURRENT codebase, unconditionally forces the
* marker-anchored checkpoint task 5 added, masking exactly the bug this
* scenario needs to demonstrate the absence of). Member A's threshold
* trips right after job1 (before job2 exists locally); member B's does not
* trip until after job2 exists locally too — modeling two members whose
* own independent checkpoint/restart timing straddles ordinary cluster
* activity happening around the upgrade. Asserts the two members' checkpoint
* files are NOT byte-identical (snapdiff_paths) and that their job SETS
* genuinely differ (job2 present in B, absent in A) — real content
* divergence, not a cosmetic LSN-header mismatch.
*
* ── Scenario 2: test_bug857_marker_checkpoint_converges_across_members ──
* The identical shared sequence, this time routed through the REAL,
* current apply_committed_entry() dispatch (persist_apply_pump_run) on
* both members. Member A catches up in a single pump-run call over the
* whole range; member B processes job1+marker in one call and only later
* commits+applies job2 in a separate second call — genuinely different
* processing pace. Asserts both members' checkpoint at the marker's own
* LSN exists and is byte-identical (snapdiff_paths) regardless of that
* pacing difference.
*/
#include <rtti.h>
#include <capnp_c.h>
#include <errno.h>
#include <inttypes.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
#include <unity/unity.h>
#include "cluster.h"
#include "config_repl.h"
#include "controller.capnp.h"
#include "hook.h"
#include "jobj.h"
#include "persist.h"
#include "snapdiff.h"
#include "wal_types.h"
#include "xce/cluster_def.h"
#include "xce/types.h"
#include "xstring.h"
/* restore_job (job.c) has no dedicated header -- every direct-call site in
* this codebase declares it locally (mirrors persist.c's own extern at the
* top of that file). This file calls it directly (bypassing
* apply_committed_entry) in Scenario 1 for the reason explained above: the
* marker's LSN must never be dispatched through apply_committed_entry
* there, or the CURRENT codebase's marker-anchored checkpoint (task 5)
* would fire and mask the exact pre-fix behavior this test reproduces. */
extern int restore_job(cluster_t* c, uint64_t jobid_unused,
const void* capnp_payload, size_t len,
uint64_t entry_lsn);
/* jobj.c's jobsub table and the process-global xstring table are shared
* across every test file in the controller_unit_test binary -- guard
* idempotently, mirroring every other config-upgrade-snapshot test file
* (job_obj_init itself is also independently idempotent via its own
* process-global atomic flag, jobj.c -- this local guard just skips the
* redundant call within this file's own repeated test invocations). */
static int g_b857_jobobj_ready;
typedef struct {
cluster_t c;
persist_t p;
char tmpdir[256];
} b857_t;
/* Fixture mirrors test_upgrade_snapshot_apply.c's tusa_t/tusa_setup exactly
* (real disk-backed persist_t + a real but otherwise-empty cluster_t with
* jobmgr/qmgr/workermgr/resmgr as xkvmgr_type instances) -- not reused
* directly since tusa_setup/tusa_t are file-static, matching this
* codebase's established convention of each test file keeping its own
* minimal local copy of such fixtures (e.g. test_replay_apply_seam.c's
* trr_make_fresh_persist mirroring test_persist_recovery.c's
* make_fresh_persist). `label` must be short and filesystem-safe -- used
* as the hostname, the persist_set_workdir name, AND the tmpdir prefix. */
static void b857_setup(b857_t* t, const char* label)
{
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 32, .delta = 16 };
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 32, .delta = 16 };
if (makeString((char*)"b857-probe") == NULL)
xstring_table_init(64);
if (!g_b857_jobobj_ready) {
TEST_ASSERT_EQUAL_INT(0, job_obj_init(32, 32));
g_b857_jobobj_ready = 1;
}
memset(t, 0, sizeof(*t));
t->c.myhost = &t->c.myhost_info;
t->c.myhost_info.name = (char*)label;
t->c.myhost_info.key = (char*)"";
TEST_ASSERT_EQUAL_INT(0,
xce_hook_init(&t->c.hooksys, XCE_CONTROLLER_HOOK_MAX));
t->c.jobmgr = makeInstance(&xkvmgr_type, &int_params);
t->c.qmgr = makeInstance(&xkvmgr_type, &str_params);
t->c.workermgr = makeInstance(&xkvmgr_type, &str_params);
t->c.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(t->c.jobmgr);
TEST_ASSERT_NOT_NULL(t->c.qmgr);
TEST_ASSERT_NOT_NULL(t->c.workermgr);
TEST_ASSERT_NOT_NULL(t->c.resmgr);
zarray_init(&t->c.sharedinst, 16, NULL);
snprintf(t->tmpdir, sizeof(t->tmpdir), "/tmp/xce_%s_XXXXXX", label);
TEST_ASSERT_NOT_NULL(mkdtemp(t->tmpdir));
persist_config_t cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000; /* slow fsync -- no background races in tests */
cfg.snap_keep_n = 4;
persist_set_workdir(&t->p, t->tmpdir, label);
TEST_ASSERT_EQUAL_INT(0, persist_init(&t->p, &cfg, &t->c));
TEST_ASSERT_EQUAL_INT(0, persist_start(&t->p));
}
/* Unconditional teardown -- callers must gather every observable they need
* to assert on BEFORE calling this (mirrors test_upgrade_snapshot_apply.c's
* documented longjmp-past-teardown discipline). */
static void b857_teardown(b857_t* t)
{
persist_stop(&t->p);
persist_fini(&t->p);
zarray_fini(&t->c.sharedinst);
if (t->c.resmgr) dereference(t->c.resmgr);
if (t->c.workermgr) dereference(t->c.workermgr);
if (t->c.qmgr) dereference(t->c.qmgr);
if (t->c.jobmgr) dereference(t->c.jobmgr);
xce_hook_fini(&t->c.hooksys);
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", t->tmpdir);
if (system(cmd) != 0)
fprintf(stderr, "cleanup warning: rm -rf %s failed\n", t->tmpdir);
}
/* Encode a minimal, real job into a WAL_JOB_SUBMIT payload -- mirrors
* test_replay_apply_seam.c's trr_encode_job exactly (capnp_encode_job
* itself is file-static in persist.c, not reusable across translation
* units, hence this small local copy). */
static int b857_encode_job(uint64_t jobid, const char* queue,
uint8_t** out, size_t* out_len)
{
job_submit_data_t submit;
memset(&submit, 0, sizeof(submit));
submit.queue = (char*)queue;
submit.user = (char*)"b857-tester";
xce_job_t j;
memset(&j, 0, sizeof(j));
j.jobid = jobid;
j.state = JOB_STAT_PEND;
j.submit = &submit;
struct capn c;
struct capn_segment* cs;
struct Job cj;
Job_ptr jp;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
encode_Job(cs, &cj, &j);
jp = new_Job(cs);
write_Job(&cj, jp);
capn_setp(capn_root(&c), 0, jp.p);
*out_len = capn_size(&c);
*out = malloc(*out_len);
if (!*out) { capn_free(&c); return -ENOMEM; }
capn_write_mem(&c, *out, *out_len, 0);
capn_free(&c);
return 0;
}
static bool b857_job_exists(var jobmgr, uint64_t jobid)
{
void* existing = NULL;
lock(jobmgr, RTTI_LOCK_READ);
int found = key_find(jobmgr, (void*)jobid, &existing);
unlock(jobmgr);
return found > 0 && existing != NULL;
}
/* Mirrors test_upgrade_snapshot_apply.c's tusa_snapshot_exists path
* construction exactly (same "snap-<lsn16>.capnp" naming convention the
* production checkpoint code uses). */
static void b857_snapshot_path(persist_t* p, uint64_t lsn, char* buf,
size_t bufsz)
{
snprintf(buf, bufsz, "%s/snap-%016" PRIx64 ".capnp", p->snap_path, lsn);
}
/* ── Scenario 1 (Req 6.2, "reproduce ... on the pre-fix code") ──────────── */
static void test_bug857_threshold_checkpoint_diverges_across_members(void)
{
b857_t A, B;
b857_setup(&A, "b857oa");
b857_setup(&B, "b857ob");
uint8_t* job1_payload = NULL; size_t job1_len = 0;
int rc_enc1 = b857_encode_job(857001, "b857q", &job1_payload, &job1_len);
uint8_t marker_sha[32];
memset(marker_sha, 0xB8, sizeof(marker_sha));
uint8_t marker_payload[WAL_CONFIG_MARKER_PAYLOAD_LEN];
size_t marker_out_len = 0;
int rc_marker_enc = config_marker_encode(0x8570000000000001ULL, marker_sha,
marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN,
&marker_out_len);
uint8_t* job2_payload = NULL; size_t job2_len = 0;
int rc_enc2 = b857_encode_job(857002, "b857q", &job2_payload, &job2_len);
/* Both members physically receive the IDENTICAL committed sequence --
* job1 @ lsn1, config-change marker @ lsn2, job2 @ lsn3 -- mirroring
* what majority replication of a real cluster guarantees. */
int rc_wal_a1 = persist_wal_async(&A.p, WAL_JOB_SUBMIT, job1_payload,
(uint32_t)job1_len);
int rc_wal_a2 = persist_wal_async(&A.p, WAL_CONFIG_MARKER, marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN);
int rc_wal_a3 = persist_wal_async(&A.p, WAL_JOB_SUBMIT, job2_payload,
(uint32_t)job2_len);
int rc_wal_b1 = persist_wal_async(&B.p, WAL_JOB_SUBMIT, job1_payload,
(uint32_t)job1_len);
int rc_wal_b2 = persist_wal_async(&B.p, WAL_CONFIG_MARKER, marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN);
int rc_wal_b3 = persist_wal_async(&B.p, WAL_JOB_SUBMIT, job2_payload,
(uint32_t)job2_len);
/* Member A's own threshold trips EARLY -- right after job1, before job2
* exists locally. Entity effect driven directly via restore_job(), the
* exact function apply_committed_entry()'s WAL_JOB_SUBMIT case itself
* calls -- the marker's LSN is never dispatched through
* apply_committed_entry() at all, so the CURRENT codebase's
* marker-anchored checkpoint (task 5) never fires here, faithfully
* reproducing the OLD, pre-task-5 decoupling: checkpoint content/timing
* driven only by schedule_checkpoint's independent WAL-record
* threshold (simulated by calling persist_checkpoint(), its real
* production entry point, directly). */
int rc_restore_a1 = restore_job(&A.c, 0, job1_payload, job1_len, 1);
atomic_store_explicit(&A.p.last_applied_lsn, 1, memory_order_relaxed);
int rc_ckpt_a = persist_checkpoint(&A.p);
uint64_t ckpt_lsn_a = A.p.checkpoint_lsn;
/* Member B's own threshold does not trip until LATER -- by then job2
* was already locally visible too (ordinary cluster activity continued
* around the upgrade before B happened to checkpoint/restart). */
int rc_restore_b1 = restore_job(&B.c, 0, job1_payload, job1_len, 1);
int rc_restore_b2 = restore_job(&B.c, 0, job2_payload, job2_len, 3);
atomic_store_explicit(&B.p.last_applied_lsn, 3, memory_order_relaxed);
int rc_ckpt_b = persist_checkpoint(&B.p);
uint64_t ckpt_lsn_b = B.p.checkpoint_lsn;
bool job1_in_a = b857_job_exists(A.c.jobmgr, 857001);
bool job1_in_b = b857_job_exists(B.c.jobmgr, 857001);
bool job2_in_a = b857_job_exists(A.c.jobmgr, 857002);
bool job2_in_b = b857_job_exists(B.c.jobmgr, 857002);
char snap_a[4096], snap_b[4096];
b857_snapshot_path(&A.p, ckpt_lsn_a, snap_a, sizeof(snap_a));
b857_snapshot_path(&B.p, ckpt_lsn_b, snap_b, sizeof(snap_b));
snapdiff_report_t rep;
memset(&rep, 0, sizeof(rep));
int diff_rc = snapdiff_paths(snap_a, snap_b, NULL, &rep);
const uint64_t marker_lsn = 2;
b857_teardown(&A);
b857_teardown(&B);
free(job1_payload);
free(job2_payload);
TEST_ASSERT_EQUAL_INT(0, rc_enc1);
TEST_ASSERT_EQUAL_INT(0, rc_marker_enc);
TEST_ASSERT_EQUAL_INT(0, rc_enc2);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a1);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a2);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a3);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b1);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b2);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b3);
TEST_ASSERT_EQUAL_INT(0, rc_restore_a1);
TEST_ASSERT_EQUAL_INT(0, rc_restore_b1);
TEST_ASSERT_EQUAL_INT(0, rc_restore_b2);
TEST_ASSERT_EQUAL_INT(0, rc_ckpt_a);
TEST_ASSERT_EQUAL_INT(0, rc_ckpt_b);
/* Neither member's checkpoint is anchored to the marker's own lsn (2)
* -- this IS the bug: "no snapshot pinning an identical post-upgrade
* state" (bug-857's own root-cause wording). */
TEST_ASSERT_EQUAL_UINT64(1, ckpt_lsn_a);
TEST_ASSERT_EQUAL_UINT64(3, ckpt_lsn_b);
TEST_ASSERT_NOT_EQUAL(marker_lsn, ckpt_lsn_a);
TEST_ASSERT_NOT_EQUAL(marker_lsn, ckpt_lsn_b);
/* Real, content-level divergence -- the JOB SET each member actually
* materialized genuinely differs, not merely a cosmetic LSN-header
* mismatch. */
TEST_ASSERT_TRUE(job1_in_a);
TEST_ASSERT_TRUE(job1_in_b);
TEST_ASSERT_FALSE_MESSAGE(job2_in_a,
"member A's checkpoint predates job2 -- must NOT contain it");
TEST_ASSERT_TRUE_MESSAGE(job2_in_b,
"member B's checkpoint postdates job2 -- must contain it");
/* And the resulting ON-DISK ARTIFACTS -- what a rejoining or
* catching-up member would actually receive via snapshot transfer --
* genuinely diverge too. */
TEST_ASSERT_EQUAL_INT(1, diff_rc);
TEST_ASSERT_FALSE_MESSAGE(rep.identical,
"pre-fix: two members' checkpoints materialized around the SAME "
"config-change marker must NOT be byte-identical -- this is the "
"bug-857 divergence, reproduced");
printf("test_bug857_threshold_checkpoint_diverges_across_members: "
"PASS\n");
}
/* ── Scenario 2 (Req 6.2, "... and pass on the fixed code") ─────────────── */
static void test_bug857_marker_checkpoint_converges_across_members(void)
{
b857_t A, B;
b857_setup(&A, "b857na");
b857_setup(&B, "b857nb");
uint8_t* job1_payload = NULL; size_t job1_len = 0;
int rc_enc1 = b857_encode_job(857101, "b857q", &job1_payload, &job1_len);
uint8_t marker_sha[32];
memset(marker_sha, 0xC9, sizeof(marker_sha));
uint8_t marker_payload[WAL_CONFIG_MARKER_PAYLOAD_LEN];
size_t marker_out_len = 0;
int rc_marker_enc = config_marker_encode(0x8570000000000002ULL, marker_sha,
marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN,
&marker_out_len);
uint8_t* job2_payload = NULL; size_t job2_len = 0;
int rc_enc2 = b857_encode_job(857102, "b857q", &job2_payload, &job2_len);
/* Member A: catches up in a SINGLE pump-run call over the whole range
* [job1, marker, job2] -- as if it processed everything in one
* uninterrupted burst. */
int rc_wal_a1 = persist_wal_async(&A.p, WAL_JOB_SUBMIT, job1_payload,
(uint32_t)job1_len);
int rc_wal_a2 = persist_wal_async(&A.p, WAL_CONFIG_MARKER, marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN);
int rc_wal_a3 = persist_wal_async(&A.p, WAL_JOB_SUBMIT, job2_payload,
(uint32_t)job2_len);
uint64_t committed_a = persist_committed_lsn_advance_atomic(&A.p, 3);
int applied_a = persist_apply_pump_run(&A.p);
/* Member B: processes job1+marker FIRST (one pump-run call, simulating
* a pause/restart boundary exactly at the marker), and only later
* commits+applies job2 in a SEPARATE second pump-run call --
* genuinely different real-world processing pace than member A. */
int rc_wal_b1 = persist_wal_async(&B.p, WAL_JOB_SUBMIT, job1_payload,
(uint32_t)job1_len);
int rc_wal_b2 = persist_wal_async(&B.p, WAL_CONFIG_MARKER, marker_payload,
WAL_CONFIG_MARKER_PAYLOAD_LEN);
uint64_t committed_b1 = persist_committed_lsn_advance_atomic(&B.p, 2);
int applied_b1 = persist_apply_pump_run(&B.p);
int rc_wal_b3 = persist_wal_async(&B.p, WAL_JOB_SUBMIT, job2_payload,
(uint32_t)job2_len);
uint64_t committed_b2 = persist_committed_lsn_advance_atomic(&B.p, 3);
int applied_b2 = persist_apply_pump_run(&B.p);
const uint64_t marker_lsn = 2;
char snap_a[4096], snap_b[4096];
b857_snapshot_path(&A.p, marker_lsn, snap_a, sizeof(snap_a));
b857_snapshot_path(&B.p, marker_lsn, snap_b, sizeof(snap_b));
bool snap_a_exists = (access(snap_a, F_OK) == 0);
bool snap_b_exists = (access(snap_b, F_OK) == 0);
snapdiff_report_t rep;
memset(&rep, 0, sizeof(rep));
int diff_rc = snapdiff_paths(snap_a, snap_b, NULL, &rep);
b857_teardown(&A);
b857_teardown(&B);
free(job1_payload);
free(job2_payload);
TEST_ASSERT_EQUAL_INT(0, rc_enc1);
TEST_ASSERT_EQUAL_INT(0, rc_marker_enc);
TEST_ASSERT_EQUAL_INT(0, rc_enc2);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a1);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a2);
TEST_ASSERT_EQUAL_INT(0, rc_wal_a3);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b1);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b2);
TEST_ASSERT_EQUAL_INT(0, rc_wal_b3);
TEST_ASSERT_EQUAL_UINT64(3, committed_a);
TEST_ASSERT_EQUAL_INT(3, applied_a);
TEST_ASSERT_EQUAL_UINT64(2, committed_b1);
TEST_ASSERT_EQUAL_INT(2, applied_b1);
TEST_ASSERT_EQUAL_UINT64(3, committed_b2);
TEST_ASSERT_EQUAL_INT(1, applied_b2);
TEST_ASSERT_TRUE_MESSAGE(snap_a_exists,
"member A must checkpoint at the marker's own lsn, even mid-batch "
"within a single pump-run call");
TEST_ASSERT_TRUE_MESSAGE(snap_b_exists,
"member B must checkpoint at the marker's own lsn, even though its "
"own processing was split across two separate pump-run calls");
/* The decisive proof: regardless of each member's own processing pace
* (A: one burst; B: paused exactly at the marker, job2 applied later),
* both marker-anchored checkpoints are byte-identical -- convergence,
* closing the bug-857 divergence Scenario 1 reproduced above. */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, diff_rc,
"post-fix: both members' marker-anchored checkpoints must be "
"byte-identical regardless of differing processing pace");
TEST_ASSERT_TRUE_MESSAGE(rep.identical,
"post-fix: materialized state converges identically across "
"members -- bug-857 no longer occurs");
printf("test_bug857_marker_checkpoint_converges_across_members: "
"PASS\n");
}
/* ── Dispatch ────────────────────────────────────────────────────────── */
void test_bug857_regression(void)
{
UnitySetTestFile(__FILE__);
printf("\n>> bug-857 materialized-state divergence regression "
"(task 8.5)\n\n");
RUN_TEST(test_bug857_threshold_checkpoint_diverges_across_members);
RUN_TEST(test_bug857_marker_checkpoint_converges_across_members);
}

177
src/controller/testsuite/test_commit_barrier_reply.c
View file

@ -66,6 +66,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
@ -3288,6 +3289,179 @@ static void test_fact_state_unchanged_throughout_quorum_loss_outage(void)
"PASS\n");
}
/* ── Batch failure propagation (raft-adapter-conformance task 5.2, Req 5.3)
* ──────────────────────────────────────────────────────────────────────
*
* persist_submit_job_payloads_async (persist.c:6135-6209) walks an xlist_t
* of xce_job_t* payloads and tracks a single `first_rc` across the loop:
* the FIRST payload whose append_job_wal_entry_local call fails sets
* first_rc and resolves that payload's completion callback (fn) with the
* failing rc (persist.c:6161-6166); every payload processed AFTER that
* point takes the `if (first_rc != 0)` branch and resolves with the SAME
* rc WITHOUT attempting its own append (persist.c:6150-6158) -- so no
* payload in a batch that has already failed can silently complete on
* its own.
*
* No existing fixture in this file or in test_commit_barrier_rollout.c
* engineers append_job_wal_entry_local/persist_wal_async itself to fail
* (the two existing XCE_TESTING fault hooks -- test_force_fsync_fail and
* test_force_job_encode_fail_at -- only guard the flusher's fdatasync and
* the SEPARATE collect_job_frames snapshot-encode loop, neither of which
* this call path reaches). This test instead forces a real filesystem
* failure: persist_init() has already mkdir_p'd persist_t's wal_path
* directory by the time it returns, so revoking write permission on that
* directory BEFORE the first append makes open_new_segment's
* open(..., O_CREAT) fail with EACCES on the very first payload --
* persist_wal_async surfaces this as -EIO (persist.c:988-992) before
* current_lsn is ever incremented. Making the FIRST payload the failing
* one exercises both branches above: payload 0 hits the original-failure
* branch, payloads 1 and 2 hit the propagation branch.
*
* A bare (cluster == NULL) persist_t is used, with no commit engine wired
* (p->repl.ce stays NULL) and persist_start() never called -- every fn()
* resolution in persist_submit_job_payloads_async's success path is then
* synchronous too (persist.c:6178-6181), so the whole batch resolves
* within the single call under test, matching the same bare-persist_t,
* no-commit-engine technique test_commit_durability.c's dur_flag_persist
* already uses for a persist_init-only fixture (persist_fini alone tears
* it down -- persist_start/persist_stop were never paired).
*
* The injected fault is made deliberately TRANSIENT, not permanent: the
* notify callback restores write permission on the WAL directory the
* instant it observes payload 0's failure, before payload 1 is ever
* processed. This is what makes the test actually exercise the
* first_rc-propagation branch rather than merely re-observing the same
* permanently-broken directory three times over: if a future change
* deleted the `if (first_rc != 0) { fn(...); continue; }` short-circuit,
* payload 1 would attempt its OWN append against a directory that is
* writable again by then, and that attempt would SUCCEED -- turning this
* test's per-payload rc assertions below into an observable failure
* instead of a vacuously-identical result. */
typedef struct {
int count;
uint64_t lsns[4];
int rcs[4];
} cbrp_bf_notify_obs_t;
/* Set by the test right after persist_init(), before the fault-injecting
* chmod(); read back by cbrp_bf_notify_fn to restore write permission the
* moment the first (and, on correct code, only genuinely-attempted)
* payload's failure is observed. */
static char g_cbrp_bf_wal_path[512];
static void cbrp_bf_notify_fn(void *ctx, uint64_t lsn, int rc)
{
cbrp_bf_notify_obs_t *obs = (cbrp_bf_notify_obs_t *)ctx;
if (obs->count == 0)
chmod(g_cbrp_bf_wal_path, 0700);
if (obs->count < 4) {
obs->lsns[obs->count] = lsn;
obs->rcs[obs->count] = rc;
}
obs->count++;
}
static job_submit_data_t g_cbrp_bf_submit;
static void test_batch_payload_persist_failure_propagates_to_whole_batch(void)
{
persist_t p;
persist_config_t cfg;
char tmpdir[256];
xlist_t plist;
uint64_t out_lsns[3] = { 0, 0, 0 };
cbrp_bf_notify_obs_t obs;
int rc;
int i;
memset(&obs, 0, sizeof(obs));
memset(&g_cbrp_bf_submit, 0, sizeof(g_cbrp_bf_submit));
g_cbrp_bf_submit.queue = (char *)"normal";
g_cbrp_bf_submit.user = (char *)"tester";
g_cbrp_bf_submit.mintasks = 1;
g_cbrp_bf_submit.maxtasks = 1;
g_cbrp_bf_submit.start_info = NULL;
g_cbrp_bf_submit.job_name = (char *)"batch-fail-job";
memset(&p, 0, sizeof(p));
cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000;
snprintf(tmpdir, sizeof(tmpdir), "/tmp/xce_cbrpbf_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmpdir));
persist_set_workdir(&p, tmpdir, "ctrl-cbrpbf");
/* cluster == NULL: append_job_wal_entry_local routes straight through
* persist_wal_async (no raft library, no commit engine involved). */
TEST_ASSERT_EQUAL_INT(0, persist_init(&p, &cfg, NULL));
/* persist_init already created p.wal_path. Revoke write permission on
* it now, before any append is attempted, so the first payload's
* open_new_segment() call fails deterministically. cbrp_bf_notify_fn
* restores it the instant payload 0's failure is observed (see the
* comment above), making the fault transient rather than permanent. */
snprintf(g_cbrp_bf_wal_path, sizeof(g_cbrp_bf_wal_path), "%s", p.wal_path);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, chmod(p.wal_path, 0500),
"test precondition: must be able to revoke write permission on "
"the WAL directory");
TEST_ASSERT_EQUAL_INT(0, xlist_init(&plist, 3));
for (i = 0; i < 3; i++) {
xce_job_t *jp = calloc(1, sizeof(*jp));
TEST_ASSERT_NOT_NULL(jp);
jp->submit = &g_cbrp_bf_submit;
jp->jobid = 70000 + (uint64_t)i;
jp->sub_time = time(NULL);
jp->queue_time = jp->sub_time;
jp->state = JOB_STAT_PEND;
TEST_ASSERT_TRUE(xlist_store(&plist, jp) >= 0);
}
rc = persist_submit_job_payloads_async(&p, &plist, out_lsns,
cbrp_bf_notify_fn, &obs);
/* Defensive fallback: cbrp_bf_notify_fn already restores write
* permission once the first payload's failure is observed (the
* normal case), but make sure the directory is writable before any
* further filesystem access (including this test's own cleanup)
* regardless of how fn was actually invoked. */
chmod(p.wal_path, 0700);
TEST_ASSERT_EQUAL_INT_MESSAGE(-EIO, rc,
"the forced WAL-directory permission failure must surface as the "
"same -EIO persist_wal_async returns on an open_new_segment "
"failure (Req 5.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(3, obs.count,
"every payload in the batch must resolve its completion callback "
"exactly once, even though none of them ever durably persisted");
for (i = 0; i < 3; i++) {
TEST_ASSERT_EQUAL_INT_MESSAGE(rc, obs.rcs[i],
"every payload -- not just the one whose append actually "
"failed -- must resolve with the SAME first-failure rc (Req "
"5.3): a future change that lets a later payload silently "
"succeed instead must fail this assertion");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, obs.lsns[i],
"a failed payload must never be resolved with a real "
"(non-zero) lsn");
}
TEST_ASSERT_EQUAL_UINT64_MESSAGE(1, p.current_lsn,
"current_lsn must not advance for any payload in a batch whose "
"very first append already failed (Req 5.2)");
for (i = 0; i < plist.count; i++) {
xce_job_t *jp = (xce_job_t *)xlist_get(&plist, i);
free(jp);
}
xlist_fini(&plist);
persist_fini(&p);
{
char cmd[300];
snprintf(cmd, sizeof(cmd), "rm -rf %s", tmpdir);
system(cmd);
}
printf("test_batch_payload_persist_failure_propagates_to_whole_batch: "
"PASS\n");
}
/* ── Test group entry point ──────────────────────────────────────────── */
void test_commit_barrier_reply(void)
@ -3356,4 +3530,7 @@ void test_commit_barrier_reply(void)
/* Durable retention held for the outage's sustained duration under
* genuine quorum loss (task 4.2, Req 6.1, 6.2) */
RUN_TEST(test_fact_state_unchanged_throughout_quorum_loss_outage);
/* Batch failure propagation regression-lock (raft-adapter-conformance
* task 5.2, Req 5.3) */
RUN_TEST(test_batch_payload_persist_failure_propagates_to_whole_batch);
}

116
src/controller/testsuite/test_commit_barrier_rollout.c
View file

@ -1505,13 +1505,18 @@ static void test_upgrade_config_barrier_failure_error_reply_no_restart(void)
printf("test_upgrade_config_barrier_failure_error_reply_no_restart: PASS\n");
}
/* Success path PIN: with a full majority of covering ACKs the existing
* behavior is unchanged — status-0 reply with the upgradeConfigReply
* payload, committed_config_tag at the saved tag, restart requested.
* Both standbys pre-ack UINT64_MAX (covers any tag — the new tag is only
* known after conf_save_config hashes the re-encoded blob), so the case
* is green under both the legacy counting and the per-push arithmetic. */
static void test_upgrade_config_majority_ack_success_unchanged(void)
/* Success path PIN: with a full majority of covering ACKs, status-0 reply
* with the upgradeConfigReply payload and committed_config_tag at the saved
* tag are unchanged — but as of config-upgrade-snapshot task 7 the
* restart/replay fallback is retired: a successful upgrade must NOT set
* restart_requested (Req 1.2/1.3/1.4/2.2/5.4). Materialization is now
* driven entirely by the marker-apply trigger (apply_committed_entry's
* WAL_CONFIG_MARKER case, persist.c, task 5), decoupled from this
* request-handling path. Both standbys pre-ack UINT64_MAX (covers any tag
* — the new tag is only known after conf_save_config hashes the re-encoded
* blob), so the case is green under both the legacy counting and the
* per-push arithmetic. */
static void test_upgrade_config_majority_ack_success_no_restart(void)
{
uint8_t *blob;
size_t blen = 0;
@ -1530,7 +1535,7 @@ static void test_upgrade_config_majority_ack_success_unchanged(void)
/* Act */
rc = handle_upgrade_config(&g_cb5_c, &g_cb5_req_ep, &g_cb5_req);
/* Assert: existing success reply + restart behavior unchanged */
/* Assert: existing success reply unchanged; restart is NOT requested */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
"a majority-acked upgrade must succeed");
cb5_scan_replies(&g_cb5_req_ep, &obs);
@ -1547,12 +1552,98 @@ static void test_upgrade_config_majority_ack_success_unchanged(void)
atomic_load_explicit(&g_cb5_c.ha.committed_config_tag,
memory_order_acquire),
"a majority-acked config must be committed at the saved tag (Req 4.1)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, atomic_load(&g_cb5_c.restart_requested),
"a successful upgrade must request the activation restart");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, atomic_load(&g_cb5_c.restart_requested),
"a successful upgrade must NOT request a restart -- materialization"
" happens via the marker-apply trigger alone (task 7, Req 1.2-1.4)");
free(blob);
cb5_teardown();
printf("test_upgrade_config_majority_ack_success_unchanged: PASS\n");
printf("test_upgrade_config_majority_ack_success_no_restart: PASS\n");
}
/* ── config-upgrade-snapshot task 7: marker-committed test hook ─────────── */
/* Captures each xce_upgrade_marker_committed_hook firing: the call count,
* the version_tag observed, and a snapshot of ha.committed_config_tag taken
* AT FIRE TIME -- proving the hook runs strictly after config_commit_change()
* has already advanced the committed tag (i.e. after the marker reached the
* committed prefix), not before. */
static int g_cb5_hook_calls;
static uint64_t g_cb5_hook_last_tag;
static uint64_t g_cb5_hook_committed_tag_at_fire;
static int cb5_marker_committed_hook(cluster_t *c, uint64_t version_tag)
{
g_cb5_hook_calls++;
g_cb5_hook_last_tag = version_tag;
g_cb5_hook_committed_tag_at_fire =
atomic_load_explicit(&c->ha.committed_config_tag, memory_order_acquire);
return 0;
}
/* The new #ifdef XCE_TESTING seam (config-upgrade-snapshot task 7,
* xce_upgrade_marker_committed_hook in config_repl.c) must fire exactly
* once per successful upgrade, right after config_commit_change() succeeds
* -- i.e. after the configuration-change marker has reached the committed
* prefix -- and must leave every other observable behavior (reply,
* committed tag, no restart) exactly as the NULL-hook case already asserts,
* proving the seam is a true no-op in production (hook defaults to NULL). */
static void test_upgrade_config_marker_committed_hook_fires_once_after_commit(void)
{
uint8_t *blob;
size_t blen = 0;
uint8_t sha[32];
int rc;
cb5_reply_obs_t obs = { 0 };
extern int (*xce_upgrade_marker_committed_hook)(cluster_t *, uint64_t);
/* Arrange: full covering pre-ack from both standbys (same fixture as
* the no-restart success-path test) */
cb5_setup_upgrade();
blob = cb5_make_config_blob(&blen);
SHA256(blob, blen, sha);
cb5_arrange_upgrade_req(blob, blen, sha);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[0], UINT64_MAX);
repl_on_config_ack(&g_cb5_c.persist.repl, &g_cb5_sb_ep[1], UINT64_MAX);
g_cb5_hook_calls = 0;
g_cb5_hook_last_tag = 0;
g_cb5_hook_committed_tag_at_fire = 0;
xce_upgrade_marker_committed_hook = cb5_marker_committed_hook;
/* Act */
rc = handle_upgrade_config(&g_cb5_c, &g_cb5_req_ep, &g_cb5_req);
/* Assert: fired exactly once, with the just-committed tag, and only
* after ha.committed_config_tag already reflects that same tag. */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "a majority-acked upgrade must succeed");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_cb5_hook_calls,
"the marker-committed test hook must fire exactly once per"
" successful upgrade (task 7)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(g_cb5_myconf.config_version_tag,
g_cb5_hook_last_tag,
"the hook must observe the marker's own version tag");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(g_cb5_myconf.config_version_tag,
g_cb5_hook_committed_tag_at_fire,
"the hook must fire AFTER config_commit_change already advanced"
" committed_config_tag, not before");
/* The rest of the observable behavior must be identical to the
* NULL-hook (production default) case -- the hook is side-effect-free
* beyond invoking the test callback itself. */
cb5_scan_replies(&g_cb5_req_ep, &obs);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, obs.n_replies,
"the requester must still receive exactly one reply with the hook set");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, obs.last_status,
"the success ACK must still carry status 0 with the hook set");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, atomic_load(&g_cb5_c.restart_requested),
"the hook firing must not resurrect the retired restart request");
xce_upgrade_marker_committed_hook = NULL;
free(blob);
cb5_teardown();
printf("test_upgrade_config_marker_committed_hook_fires_once_after_commit:"
" PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
@ -2351,7 +2442,8 @@ void test_commit_barrier_rollout(void)
RUN_TEST(test_push_config_withheld_acks_returns_etimedout);
RUN_TEST(test_push_config_majority_acks_returns_zero_and_commits);
RUN_TEST(test_upgrade_config_barrier_failure_error_reply_no_restart);
RUN_TEST(test_upgrade_config_majority_ack_success_unchanged);
RUN_TEST(test_upgrade_config_majority_ack_success_no_restart);
RUN_TEST(test_upgrade_config_marker_committed_hook_fires_once_after_commit);
/* Task 5.2 remediation r1 — production wire-path ACK ingestion
* (completes the unlanded half of ha-config-bootstrap task 8.1) */
RUN_TEST(test_config_ack_wire_path_completes_barrier);

3247
src/controller/testsuite/test_config_repl.c
View file

File diff suppressed because it is too large Load diff

217
src/controller/testsuite/test_configcompare.c Normal file
View file

@ -0,0 +1,217 @@
#define _GNU_SOURCE
#include <errno.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <unity/unity.h>
#include "configcompare.h"
static char g_dir[256];
static void fresh_dir(void)
{
strcpy(g_dir, "/tmp/xce_configcompare_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(g_dir));
}
static void cleanup_dir(void)
{
char cmd[512];
snprintf(cmd, sizeof(cmd), "rm -rf %s", g_dir);
system(cmd);
}
/* Creates <root>/<name>/conf/ (a member's `workdir/name` directory, mirroring
* conf_dir = "%s/%s", myconf->workdir, myconf->name in persist.c /
* config_repl.c) and writes the resulting path into out_path. Does NOT touch
* config_repl.c or persist.c -- these are hand-written fixture files so the
* test exercises configcompare.c's OWN resolve-then-read logic, not a
* different code path's. */
static void make_member_dir(const char* root, const char* name,
char out_path[512])
{
snprintf(out_path, 512, "%s/%s", root, name);
TEST_ASSERT_EQUAL_INT(0, mkdir(out_path, 0700));
char conf_dir[560];
snprintf(conf_dir, sizeof(conf_dir), "%s/conf", out_path);
TEST_ASSERT_EQUAL_INT(0, mkdir(conf_dir, 0700));
}
static void write_conf_current(const char* member_path, const char* hexsuffix)
{
char ptr_path[600];
snprintf(ptr_path, sizeof(ptr_path), "%s/conf/current", member_path);
FILE* f = fopen(ptr_path, "w");
TEST_ASSERT_NOT_NULL(f);
fprintf(f, "cluster.bin.%s\n", hexsuffix);
fclose(f);
}
static void write_conf_blob(const char* member_path, const char* hexsuffix,
const uint8_t* bytes, size_t len)
{
char blob_path[600];
snprintf(blob_path, sizeof(blob_path), "%s/conf/cluster.bin.%s",
member_path, hexsuffix);
FILE* f = fopen(blob_path, "wb");
TEST_ASSERT_NOT_NULL(f);
if (len > 0)
TEST_ASSERT_EQUAL_INT((int)len, (int)fwrite(bytes, 1, len, f));
fclose(f);
}
/* Full happy-path member: dir + conf/current + the versioned blob it
* points at. */
static void setup_member_ok(const char* root, const char* name,
const char* hexsuffix, const uint8_t* bytes,
size_t len, char out_path[512])
{
make_member_dir(root, name, out_path);
write_conf_current(out_path, hexsuffix);
write_conf_blob(out_path, hexsuffix, bytes, len);
}
static void test_two_members_identical_config_reports_identical(void)
{
fresh_dir();
const uint8_t data[] = "identical config bytes for both members";
char m0[512], m1[512];
setup_member_ok(g_dir, "m0", "aaaaaaaaaaaa", data, sizeof(data) - 1, m0);
setup_member_ok(g_dir, "m1", "bbbbbbbbbbbb", data, sizeof(data) - 1, m1);
const char* paths[2] = { m0, m1 };
configcompare_report_t rep;
int rc = configcompare_paths(paths, 2, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.all_present);
TEST_ASSERT_EQUAL_INT(1, rep.all_identical);
TEST_ASSERT_EQUAL_INT(-1, rep.first_divergent_index);
cleanup_dir();
}
static void test_two_members_differing_config_reports_divergent(void)
{
fresh_dir();
const uint8_t data0[] = "config bytes version A";
const uint8_t data1[] = "config bytes version B, quite different";
char m0[512], m1[512];
setup_member_ok(g_dir, "m0", "aaaaaaaaaaaa", data0, sizeof(data0) - 1, m0);
setup_member_ok(g_dir, "m1", "bbbbbbbbbbbb", data1, sizeof(data1) - 1, m1);
const char* paths[2] = { m0, m1 };
configcompare_report_t rep;
int rc = configcompare_paths(paths, 2, stdout, &rep);
TEST_ASSERT_EQUAL_INT(1, rc);
TEST_ASSERT_EQUAL_INT(1, rep.all_present);
TEST_ASSERT_EQUAL_INT(0, rep.all_identical);
TEST_ASSERT_EQUAL_INT(1, rep.first_divergent_index);
cleanup_dir();
}
/* A member with no conf/current pointer file at all must be reported as
* unreadable (read_rc != 0), never silently folded into "identical" or
* "divergent" against the two members that DO agree. */
static void test_missing_conf_current_reported_unreadable_not_divergent(void)
{
fresh_dir();
const uint8_t data[] = "shared config bytes across the healthy members";
char m0[512], m1[512], m2[512];
setup_member_ok(g_dir, "m0", "aaaaaaaaaaaa", data, sizeof(data) - 1, m0);
/* m1: conf/ dir exists but conf/current was never written. */
make_member_dir(g_dir, "m1", m1);
setup_member_ok(g_dir, "m2", "cccccccccccc", data, sizeof(data) - 1, m2);
const char* paths[3] = { m0, m1, m2 };
configcompare_report_t rep;
int rc = configcompare_paths(paths, 3, stdout, &rep);
TEST_ASSERT_LESS_THAN_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, rep.read_rc[0]);
TEST_ASSERT_NOT_EQUAL(0, rep.read_rc[1]);
TEST_ASSERT_EQUAL_INT(0, rep.all_present);
cleanup_dir();
}
/* A member whose conf/current points at a versioned file that does not
* exist on disk gets the same "unreadable" treatment as a missing pointer
* file -- neither is evidence of matching or mismatching configuration. */
static void test_current_points_to_missing_versioned_file_reported_unreadable(void)
{
fresh_dir();
const uint8_t data[] = "shared config bytes, dangling pointer case";
char m0[512], m1[512];
setup_member_ok(g_dir, "m0", "aaaaaaaaaaaa", data, sizeof(data) - 1, m0);
/* m1: conf/current points at cluster.bin.dddddddddddd, but that
* versioned file was never written. */
make_member_dir(g_dir, "m1", m1);
write_conf_current(m1, "dddddddddddd");
const char* paths[2] = { m0, m1 };
configcompare_report_t rep;
int rc = configcompare_paths(paths, 2, stdout, &rep);
TEST_ASSERT_LESS_THAN_INT(0, rc);
TEST_ASSERT_EQUAL_INT(0, rep.read_rc[0]);
TEST_ASSERT_NOT_EQUAL(0, rep.read_rc[1]);
cleanup_dir();
}
/* Basic determinism sanity check: comparing the same two byte-identical
* files twice in a row yields the same hash both times. */
static void test_repeated_comparison_of_identical_files_is_deterministic(void)
{
fresh_dir();
const uint8_t data[] = "deterministic hash check bytes, fixed content";
char m0[512], m1[512];
setup_member_ok(g_dir, "m0", "aaaaaaaaaaaa", data, sizeof(data) - 1, m0);
setup_member_ok(g_dir, "m1", "bbbbbbbbbbbb", data, sizeof(data) - 1, m1);
const char* paths[2] = { m0, m1 };
configcompare_report_t rep1, rep2;
int rc1 = configcompare_paths(paths, 2, NULL, &rep1);
int rc2 = configcompare_paths(paths, 2, NULL, &rep2);
TEST_ASSERT_EQUAL_INT(0, rc1);
TEST_ASSERT_EQUAL_INT(0, rc2);
TEST_ASSERT_EQUAL_INT(0, memcmp(rep1.sha256[0], rep2.sha256[0], 32));
TEST_ASSERT_EQUAL_INT(0, memcmp(rep1.sha256[1], rep2.sha256[1], 32));
cleanup_dir();
}
/* Fewer than two members means there is nothing to compare -- a negative
* errno, not a degenerate "identical" verdict. */
static void test_n_less_than_2_returns_negative_errno(void)
{
const char* one_path[1] = { "/tmp/xce_configcompare_irrelevant" };
configcompare_report_t rep;
int rc0 = configcompare_paths(NULL, 0, NULL, &rep);
TEST_ASSERT_LESS_THAN_INT(0, rc0);
int rc1 = configcompare_paths(one_path, 1, NULL, &rep);
TEST_ASSERT_LESS_THAN_INT(0, rc1);
}
void test_configcompare(void)
{
UnitySetTestFile(__FILE__);
RUN_TEST(test_two_members_identical_config_reports_identical);
RUN_TEST(test_two_members_differing_config_reports_divergent);
RUN_TEST(test_missing_conf_current_reported_unreadable_not_divergent);
RUN_TEST(test_current_points_to_missing_versioned_file_reported_unreadable);
RUN_TEST(test_repeated_comparison_of_identical_files_is_deterministic);
RUN_TEST(test_n_less_than_2_returns_negative_errno);
}

27
src/controller/testsuite/test_convergence_verification.c
View file

@ -276,17 +276,22 @@ static int cv_standby_in_catch_up(repl_ep_t ep)
/* ── Tests ──────────────────────────────────────────────────────────────── */
/* 5.1: a walAck with acked_lsn < target = max(committed_lsn, last_written)
* schedules EXACTLY ONE continuation that streams [acked_lsn+1, target] without
* any intervening heartbeat/stateReport dispatch.
/* 5.1: a walAck with acked_lsn < target = ha_compute_repl_target_lsn(
* committed_lsn, last_written) schedules EXACTLY ONE continuation that
* streams [acked_lsn+1, target] without any intervening heartbeat/
* stateReport dispatch.
* RED: the walAck continuation scaffold schedules nothing → no task, the stream
* hook is never called. */
static void test_walack_behind_target_schedules_one_continuation(void)
{
/* appended_lsn=9 → last_written=9; committed_lsn=4 → target=max(4,9)=9. */
/* appended_lsn=9 → last_written=9; committed_lsn=9 → target=9. Kept equal
* to last_written (rather than a lower value like the historical 4) so
* this test exercises catch-up continuation scheduling only — the
* committed_lsn-vs-last_written edge case itself (bug-1478) has its own
* dedicated pure-function coverage in test_ha_role.c. */
cv_setup("behind", /*n_peers=*/3, /*n_standbys=*/1,
/*appended_lsn=*/9, /*n_vcpus=*/8);
atomic_store(&g_cv.persist.committed_lsn, 4);
atomic_store(&g_cv.persist.committed_lsn, 9);
const int peer_idx = 0;
const uint64_t acked_lsn = 3; /* behind target 9 */
@ -325,8 +330,8 @@ static void test_walack_behind_target_schedules_one_continuation(void)
"[acked_lsn+1, target]) (Req 5.1)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(target,
atomic_load(&g_stream_last_target_lsn),
"the continuation must stream up to target = max(committed_lsn, "
"last_written) (Req 5.1)");
"the continuation must stream up to target = "
"ha_compute_repl_target_lsn(committed_lsn, last_written) (Req 5.1)");
cv_teardown();
printf("test_walack_behind_target_schedules_one_continuation: checked\n");
@ -338,9 +343,11 @@ static void test_walack_behind_target_schedules_one_continuation(void)
* "exactly one continuation across two ACKs" assertion fails (0 != 1). */
static void test_walack_single_flight_no_duplicate_continuation(void)
{
/* committed_lsn == appended_lsn (target=9 either way) — see the "behind"
* test's own note on why these don't use a lower committed_lsn. */
cv_setup("single", /*n_peers=*/3, /*n_standbys=*/1,
/*appended_lsn=*/9, /*n_vcpus=*/8);
atomic_store(&g_cv.persist.committed_lsn, 4);
atomic_store(&g_cv.persist.committed_lsn, 9);
const int peer_idx = 0;
@ -382,9 +389,11 @@ static void test_walack_single_flight_no_duplicate_continuation(void)
* RED: the scaffold never schedules/clears, so neither continuation appears. */
static void test_walack_reschedules_after_continuation_completes(void)
{
/* committed_lsn == appended_lsn (target=9 either way) — see the "behind"
* test's own note on why these don't use a lower committed_lsn. */
cv_setup("respawn", /*n_peers=*/3, /*n_standbys=*/1,
/*appended_lsn=*/9, /*n_vcpus=*/8);
atomic_store(&g_cv.persist.committed_lsn, 4);
atomic_store(&g_cv.persist.committed_lsn, 9);
const int peer_idx = 0;

243
src/controller/testsuite/test_ha_core_safety.c
View file

@ -2645,6 +2645,234 @@ static void test_6_2_real_senders_never_produce_termblind_shape(void)
printf("test_6_2_real_senders_never_produce_termblind_shape: done\n");
}
/* Task 4.1 (raft-adapter-conformance, Requirement 3.3) sanity check: this is
* NOT task 4.2's "drives the production heartbeat/replication path
* end-to-end and asserts the counter stays at zero" non-regression test —
* that test belongs to task 4.2 and lives separately. This is only a
* trivial proof that the counter itself works: it is directly
* readable/resettable from a test (XCE_TESTING extern linkage, ha.h) and
* increments by exactly one per invocation of ha_cb_send_append_entries_stub
* (driven here via the existing ha_cb_send_append_entries_for_test seam).
* ha==NULL still reaches the increment — it is the very first statement in
* the callback body, before the ha/raft_c NULL guard — so this needs no
* cluster/persist fixture at all. */
static void test_4_1_send_append_entries_stub_call_count_increments(void)
{
ha_send_append_entries_stub_call_count = 0;
int rc = ha_cb_send_append_entries_for_test(NULL, /*peer*/0,
/*prev_idx*/0, /*prev_term*/0,
/*leader_commit*/0, /*n*/0);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(1, ha_send_append_entries_stub_call_count,
"ha_send_append_entries_stub_call_count must increment once per "
"ha_cb_send_append_entries_stub invocation");
rc = ha_cb_send_append_entries_for_test(NULL, /*peer*/0, 0, 0, 0, 0);
TEST_ASSERT_EQUAL_INT(-EINVAL, rc);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(2, ha_send_append_entries_stub_call_count,
"the counter must accumulate across multiple invocations, not just "
"latch at 1");
printf("test_4_1_send_append_entries_stub_call_count_increments: PASS\n");
}
/* Task 4.2 (raft-adapter-conformance, Requirement 3.3): the fuller
* non-regression guard task 4.1's doc-comment promises — drive BOTH real
* production AppendEntries-equivalent senders (repl_broadcast for live
* in-term replication, ha_stream_wal_to_standby for catch-up streaming) end
* to end and assert ha_send_append_entries_stub_call_count stays at 0
* throughout, while independently confirming each real sender actually fired
* for this scenario — so the zero-count assertion is not vacuously true
* because nothing ran. Reuses the exact fixtures/hooks task 6.2's
* test_6_2_real_senders_never_produce_termblind_shape already established for
* driving these same two real senders (repl_broadcast_send_hook +
* hcs_6_2_broadcast_hook/g_hcs62_bcast_n for sender 1;
* xce_wal_stream_request_hook + hcs_2_3_capture_hook/g_sends_n for sender 2).
* If a future change ever gives raft_send_append_entries a production
* caller, this test catches it. */
static void test_4_2_dormant_sender_stays_uncalled_in_production(void)
{
/* Reset first: an earlier test in this same process (task 4.1's own
* sanity test, which runs immediately before this one in the suite)
* already drove ha_cb_send_append_entries_for_test directly and left the
* counter non-zero — Unity tests share process state. */
ha_send_append_entries_stub_call_count = 0;
/* ── Real sender 1: repl_broadcast (live in-term replication) ─────────── */
replicator_t r;
TEST_ASSERT_EQUAL_INT(0, repl_init(&r));
int dummy_ep = 0;
TEST_ASSERT_TRUE(repl_add_standby(&r, (void*)&dummy_ep, "standby", 7101) >= 0);
uint8_t full[HCS_FULL_LEN];
hcs_fill_body(full);
wal_entry_hdr_t hdr;
memset(&hdr, 0, sizeof(hdr));
hdr.lsn = 1;
hdr.type = WAL_JOB_SUBMIT;
hdr.term = 3;
g_hcs62_bcast_n = 0;
repl_broadcast_send_hook = hcs_6_2_broadcast_hook;
int brc = repl_broadcast(&r, &hdr, full, HCS_FULL_LEN, /*prev_log_term=*/0);
repl_broadcast_send_hook = NULL;
repl_fini(&r);
TEST_ASSERT_EQUAL_INT(0, brc);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_hcs62_bcast_n,
"repl_broadcast must have actually sent to the registered standby — "
"otherwise the counter-stays-zero assertion below would be vacuous");
/* ── Real sender 2: ha_stream_wal_to_standby (catch-up streaming) ──────── */
hcs_member_t src, rcv;
TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&src, "42src"));
TEST_ASSERT_EQUAL_INT(0, hcs_member_up(&rcv, "42rcv"));
TEST_ASSERT_EQUAL_INT(0, hcs_wal_put(&src, 1, /*term=*/3, WAL_JOB_SUBMIT,
full, HCS_FULL_LEN));
atomic_store_explicit(&src.p->committed_lsn, 1, memory_order_release);
hcs_2_3_reset(&rcv, /*fault_lsn=*/0 /* no fault injected — happy path */);
xce_wal_stream_request_hook = hcs_2_3_capture_hook;
int src_rc = ha_stream_wal_to_standby((endpoint_t*)NULL, src.wal_dir, src.p,
/*leader_term=*/3, /*standby_lsn=*/0,
/*committed_lsn=*/1);
xce_wal_stream_request_hook = NULL;
TEST_ASSERT_EQUAL_INT(0, src_rc);
TEST_ASSERT_TRUE_MESSAGE(g_sends_n >= 1,
"ha_stream_wal_to_standby must have actually streamed at least one "
"entry — otherwise the counter-stays-zero assertion below would be "
"vacuous");
hcs_member_down(&src);
hcs_member_down(&rcv);
/* ── The Requirement 3.3 non-regression guard itself ────────────────────
* Both real production AppendEntries-equivalent senders just fired
* (proven non-vacuous above) — yet the dormant
* ha_cb_send_append_entries_stub callback, reachable only via
* raft_send_append_entries (zero production callers), must never have
* been invoked by driving this real path. */
TEST_ASSERT_EQUAL_UINT64_MESSAGE(0, ha_send_append_entries_stub_call_count,
"driving the production heartbeat/replication path end-to-end must "
"NEVER increment ha_send_append_entries_stub_call_count — a nonzero "
"count here means raft_send_append_entries acquired a production "
"caller, invalidating the Requirement 3.3 'documented dormant' "
"resolution");
printf("test_4_2_dormant_sender_stays_uncalled_in_production: PASS\n");
}
/* ═══════════════════════════════════════════════════════════════════════════
* TASK 5.1 (raft-adapter-conformance) — regression-lock: a WAL catch-up
* rejection actually sends the NACK reply on the wire.
*
* bug-1479 fixed ha_apply_term_entry_via_library's logOk-reject branch (ha.c
* ~3315-3327) to call ha_send_wal_ack_reply_ex(ha, ep, c, e->lsn, success=0)
* instead of silently dropping the reject — before that fix NOTHING was sent
* on reject, so the primary's NACK-handling (ha_raft_on_repl_nack) was correct
* but unreachable dead code. Downstream tests (ha_raft_on_repl_nack's own
* suite, catch-up-triggering tests) already prove the EFFECTS of a NACK once
* one arrives; this test closes the reply-CONSTRUCTION-boundary gap itself —
* that the reject branch actually invokes ha_send_wal_ack_reply_ex(success=0)
* — asserted here via the ha_wal_ack_send_hook seam (ha.h, XCE_TESTING),
* mirroring test_config_repl.c's test_bug1479_ae_reject_sends_nack_not_silence
* but built from THIS file's own hcs4_* fixture family (Requirement 2.1's
* _Boundary:_ is this file) so a future change to the reject branch is caught
* here too, independent of test_config_repl.c.
*
* Fixture: a STANDBY follower with a real persisted LSN1@STORED_TERM tip. A
* term-carrying single-entry AppendEntries claims prev=(1, CLAIMED_PREV_TERM)
* — a genuine (term,lsn) mismatch against the real on-disk term at LSN1 — so
* logOk fails and ha_apply_term_entry_via_library takes the reject branch.
* Driven through the REAL production ingest seam, ha_apply_wal_catchup_entry
* (the exact function every incoming WAL replication message is routed to),
* the same seam task 6.1 already established in this file.
* ═══════════════════════════════════════════════════════════════════════════ */
static uint64_t g_hcs51_hook_term;
static uint64_t g_hcs51_hook_lsn;
static int g_hcs51_hook_success;
static int g_hcs51_hook_called;
static void hcs51_wal_ack_capture(uint64_t term, uint64_t lsn, int success)
{
g_hcs51_hook_called = 1;
g_hcs51_hook_term = term;
g_hcs51_hook_lsn = lsn;
g_hcs51_hook_success = success;
}
static void test_5_1_wal_catchup_reject_sends_nack_reply(void)
{
cluster_t* c = hcs4_cluster_up("wal_nack");
TEST_ASSERT_NOT_NULL(c);
enum { FOLLOWER_TERM = 4, STORED_TERM = 2, CLAIMED_PREV_TERM = 3 };
hcs4_make_follower(c, /*term*/FOLLOWER_TERM, /*n_peers*/3);
/* Real persisted tip: LSN1 @ STORED_TERM, via the wired standby persist
* callback (real fs required for inline fdatasync — bug-924). */
uint8_t full[HCS_FULL_LEN];
hcs_fill_body(full);
uint8_t blob[sizeof(raft_entry_blob_t) + HCS_FULL_LEN];
size_t n = hcs4_pack_entry(blob, WAL_JOB_SUBMIT, full, HCS_FULL_LEN);
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
ha_cb_persist_entry_for_test(&c->ha, 1, STORED_TERM, blob, n),
"seed write must persist (real fs required — bug-924)");
c->ha.raft.log_len = 1;
/* A term-carrying entry at LSN2 claims prev=(1, CLAIMED_PREV_TERM) — a
* genuine (term,lsn) mismatch against the real LSN1@STORED_TERM on disk
* (CLAIMED_PREV_TERM != STORED_TERM), so logOk MUST reject it. */
uint8_t lpay[1] = { 'z' };
wal_catchup_entry_t sub;
memset(&sub, 0, sizeof(sub));
sub.lsn = 2;
sub.entry_type = WAL_JOB_UPDATE;
sub.term = FOLLOWER_TERM;
sub.prev_log_index = 1;
sub.prev_log_term = CLAIMED_PREV_TERM;
sub.raw_payload = lpay;
sub.raw_payload_len = 1;
wal_catchup_entry_t* sub_arr[1] = { &sub };
wal_catchup_entry_t top;
memset(&top, 0, sizeof(top));
top.n_entries = 1;
top.entries = sub_arr;
g_hcs51_hook_called = 0;
ha_wal_ack_send_hook = hcs51_wal_ack_capture;
int rc = ha_apply_wal_catchup_entry(&c->ha, &c->persist, &top,
/*ep*/NULL, c);
ha_wal_ack_send_hook = NULL; /* uninstall before any assertion can fail */
TEST_ASSERT_EQUAL_INT(0, rc); /* reject, not a hard error */
/* Nothing was accepted at LSN2: the rejected entry never landed. */
hcs_probe_t pr;
TEST_ASSERT_EQUAL_INT(0, hcs4_probe(c, 2, &pr));
TEST_ASSERT_FALSE_MESSAGE(pr.present,
"a rejected entry must never be persisted at its LSN");
TEST_ASSERT_TRUE_MESSAGE(g_hcs51_hook_called,
"a WAL catch-up rejection must send a WalAckReply NACK "
"(ha_send_wal_ack_reply_ex success=0), not silence it -- the "
"reply-construction boundary bug-1479 fixed");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_hcs51_hook_success,
"the reply for a rejected WAL catch-up entry must carry success=0");
TEST_ASSERT_EQUAL_UINT64((uint64_t)FOLLOWER_TERM, g_hcs51_hook_term);
TEST_ASSERT_EQUAL_UINT64(2, g_hcs51_hook_lsn);
hcs4_cluster_down(c);
printf("test_5_1_wal_catchup_reject_sends_nack_reply: PASS\n");
}
/* ── Suite entry point ─────────────────────────────────────────────────────*/
void test_ha_core_safety(void)
@ -2740,4 +2968,19 @@ void test_ha_core_safety(void)
* above and in ha.c. */
RUN_TEST(test_6_2_restart_preserves_term_identity);
RUN_TEST(test_6_2_real_senders_never_produce_termblind_shape);
/* Task 4.1 (raft-adapter-conformance) sanity check: the new
* ha_send_append_entries_stub_call_count non-regression-guard counter
* itself increments correctly. Task 4.2 owns the fuller "stays at zero
* across a real production replication drive" assertion. */
RUN_TEST(test_4_1_send_append_entries_stub_call_count_increments);
/* Task 4.2 (raft-adapter-conformance): the fuller non-regression guard —
* drive repl_broadcast and ha_stream_wal_to_standby (the two real
* production senders) end-to-end and confirm the dormant stub's counter
* never moves off zero, while proving each real sender actually fired. */
RUN_TEST(test_4_2_dormant_sender_stays_uncalled_in_production);
/* Task 5.1 (raft-adapter-conformance): regression-lock the bug-1479
* reply-construction boundary — a WAL catch-up rejection must actually
* invoke ha_send_wal_ack_reply_ex(success=0) via ha_wal_ack_send_hook,
* not silently drop the reject. */
RUN_TEST(test_5_1_wal_catchup_reject_sends_nack_reply);
}

123
src/controller/testsuite/test_ha_role.c
View file

@ -660,6 +660,119 @@ static void test_evaluate_repl_action_stream_wal_fresh_system(void)
printf("test_evaluate_repl_action_stream_wal_fresh_system: PASS\n");
}
/* ── bug-1478: ha_compute_repl_target_lsn ─────────────────────────────────
*
* repl_ack_decode_and_dispatch (ha.c) computes the replication/convergence
* target passed into ha_dispatch_repl_action_for_standby as
* max(committed_lsn, last_written), where last_written is the raw locally-
* appended WAL tip (persist.current_lsn - 1). A live 3-node HA soak
* (config-upgrade-snapshot task 8.9, --config-upgrade-wal-trim) caught this
* unconditional max() permanently wedging catch-up: once a marker-triggered
* checkpoint (a local-only, NEVER-replicated WAL_CHECKPOINT entry —
* persist_write_checkpoint_marker bumps current_lsn directly, bypassing
* raft_log_append/the commit engine, bug-1466) trails an already-advancing
* committed_lsn, last_written sits permanently one LSN past it — a position
* no standby's applied_lsn can EVER legitimately reach (the checkpoint's own
* slot is, by design, never streamed or applied). ha_evaluate_repl_action's
* standby_lsn < committed_lsn check then never reaches HA_REPL_CONVERGED,
* and the primary re-streams the same empty range every heartbeat, forever.
* Ground-truthed via `xce_test waldump` on the live soak's preserved WAL
* (ctrl-2: lsn 21 = the WAL_CONFIG_MARKER, lsn 22 = WAL_CHECKPOINT, 0 bytes)
* and via ctrl-2's own commit_advance_locked log (the REAL, majority-gated
* committed_lsn never advanced past 21 — only the target_lsn fallback
* leaked 22 into the heartbeat decision).
*/
static void test_compute_repl_target_lsn_bootstrap_fallback(void)
{
/* committed_lsn genuinely stuck at 0 (no standby has ever acked) — the
* documented bootstrap exception: fall back to last_written so a
* first-ever connecting standby can catch up to real data. */
TEST_ASSERT_EQUAL_UINT64(42, ha_compute_repl_target_lsn(0, 42));
printf("test_compute_repl_target_lsn_bootstrap_fallback: PASS\n");
}
static void test_compute_repl_target_lsn_zero_zero(void)
{
/* Genuinely fresh cluster, nothing appended yet. */
TEST_ASSERT_EQUAL_UINT64(0, ha_compute_repl_target_lsn(0, 0));
printf("test_compute_repl_target_lsn_zero_zero: PASS\n");
}
static void test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn(void)
{
/* bug-1478's exact reproduction shape: committed_lsn=21 is already
* advancing (non-zero, real, majority-gated), but last_written=22
* reflects a local-only WAL_CHECKPOINT that trails it. The bootstrap
* fallback must NOT fire here — the target must stay at the real
* committed_lsn (21), never the unstreamable checkpoint slot (22). */
TEST_ASSERT_EQUAL_UINT64(21, ha_compute_repl_target_lsn(21, 22));
printf("test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn: PASS\n");
}
static void test_compute_repl_target_lsn_committed_already_ahead(void)
{
/* Normal steady-state case: committed_lsn is already the higher/current
* value; last_written (a possibly-stale, lock-free read) trails it.
* The target must simply be committed_lsn. */
TEST_ASSERT_EQUAL_UINT64(100, ha_compute_repl_target_lsn(100, 50));
printf("test_compute_repl_target_lsn_committed_already_ahead: PASS\n");
}
/* ── bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn ─────────────────
*
* Distinct from ha_compute_repl_target_lsn above (the CONVERGENCE target,
* bootstrap-only fallback). This is the STREAM ceiling: a live 3-node HA
* soak (config-upgrade-snapshot task 8.6, force-elect round) caught
* committed_lsn-bounded streaming in a permanent deadlock -- with one member
* down and the other standby log-conflicted after a chaotic election, that
* standby's own ack is the ONLY path to a new majority, but it can never
* receive any entry past committed_lsn to ack in the first place, since
* ha_stream_wal_to_standby refuses to send anything beyond it. The ceiling
* must extend to the leader's real log tail so catch-up can make progress
* past a stalled commit point, while still excluding a trailing local-only
* checkpoint LSN (the exact bug-1478 trap) since no standby can ever
* legitimately ack that exact slot either. */
static void test_compute_repl_stream_ceiling_lsn_widens_past_committed(void)
{
/* The core bug-1479 gap-3 fix: committed_lsn is stalled at 14, but the
* leader has real, streamable entries up to 33 -- the ceiling must
* widen to 33, not stay capped at 14. */
TEST_ASSERT_EQUAL_UINT64(33,
ha_compute_repl_stream_ceiling_lsn(14, 33, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_widens_past_committed: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint(void)
{
/* bug-1478's exact trap, reintroduced here if unguarded: last_written
* (15) IS the trailing local-only checkpoint's own consumed LSN slot
* (checkpoint_lsn=15) -- no standby can ever legitimately ack that exact
* value, so the ceiling must fall back to committed_lsn (14), not widen
* to the unreachable checkpoint slot. */
TEST_ASSERT_EQUAL_UINT64(14,
ha_compute_repl_stream_ceiling_lsn(14, 15, /*checkpoint_lsn*/15));
printf("test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_committed_already_ahead(void)
{
/* Normal steady state: committed_lsn is already the higher value (a
* standby that hasn't caught up to a recently-advanced commit). The
* ceiling must simply be committed_lsn, never go backward. */
TEST_ASSERT_EQUAL_UINT64(100,
ha_compute_repl_stream_ceiling_lsn(100, 50, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_committed_already_ahead: PASS\n");
}
static void test_compute_repl_stream_ceiling_lsn_zero_zero(void)
{
/* Genuinely fresh cluster, nothing appended yet. */
TEST_ASSERT_EQUAL_UINT64(0,
ha_compute_repl_stream_ceiling_lsn(0, 0, /*checkpoint_lsn*/0));
printf("test_compute_repl_stream_ceiling_lsn_zero_zero: PASS\n");
}
/* The transitional-leader timeout default test was removed: the config knob
* and its default macro are deleted by ha-single-leader-role (the wall-clock
* force-promotion watchdog it configured is gone — Req 1.5). */
@ -892,6 +1005,16 @@ void test_ha_role(void)
RUN_TEST(test_evaluate_repl_action_stream_wal);
/* Task 6.1: (tag=T, lsn=0) with oldest_wal_lsn=0 → STREAM_WAL */
RUN_TEST(test_evaluate_repl_action_stream_wal_fresh_system);
/* bug-1478: ha_compute_repl_target_lsn */
RUN_TEST(test_compute_repl_target_lsn_bootstrap_fallback);
RUN_TEST(test_compute_repl_target_lsn_zero_zero);
RUN_TEST(test_compute_repl_target_lsn_ignores_trailing_checkpoint_lsn);
RUN_TEST(test_compute_repl_target_lsn_committed_already_ahead);
/* bug-1479 (gap 3): ha_compute_repl_stream_ceiling_lsn */
RUN_TEST(test_compute_repl_stream_ceiling_lsn_widens_past_committed);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_excludes_trailing_checkpoint);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_committed_already_ahead);
RUN_TEST(test_compute_repl_stream_ceiling_lsn_zero_zero);
/* Task 3.3: ha_dispatch_repl_action_for_standby PUSH_CONFIG wiring */
RUN_TEST(test_dispatch_repl_action_push_config_when_tag_behind);
RUN_TEST(test_dispatch_repl_action_no_push_when_converged);

235
src/controller/testsuite/test_ha_vote_send.c
View file

@ -654,6 +654,236 @@ static void test_bug1417_middle_index_real_vote_from_peer2_not_dropped(void)
printf("test_bug1417_middle_index_real_vote_from_peer2_not_dropped: PASS\n");
}
/* ── raft-adapter-conformance task 3.1: post-connect vote failure routing ──
*
* vote_ep_err_handler's election-phase branch (the iomgr err_handler fired
* when an already-connected outgoing vote endpoint fails/closes before a
* reply arrives) must count that outcome through vote_reply_dispatch -- the
* SAME production tally path a connect-time failure (raft_cb_send_vote_request's
* own rc<0 branches, bug-1423) or a delivered denial (vote_reply_msg_handler)
* uses -- not the legacy handle_request_vote_reply, which bypasses
* raft_on_vote_reply entirely and, critically for this test, never reaches the
* xce_vote_reply_hook seam vote_reply_dispatch checks.
*
* Test seam: xce_vote_reply_hook (checked ONLY inside vote_reply_dispatch) lets
* us observe, without a live iomgr/raft election, whether the post-connect
* failure reached the production tally path. Pre-fix, this hook is never
* invoked by vote_ep_err_handler's election branch -- proving the bug.
*/
extern int vote_ep_err_handler_for_test(endpoint_t *ep);
static void test_post_connect_vote_failure_routes_through_production_tally(void)
{
reset_reply_mock();
ha_state_t ha;
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &g_vote_cfg, 0));
/* Election phase, not CTRL_PRIMARY -- exercises the "else" branch. */
atomic_store_explicit(&ha.role, CTRL_ELECTING, memory_order_release);
atomic_store_explicit(&ha.term, 4ULL, memory_order_release);
ha_elect_ctx_t *ctx = alloc_ctx_for_vote(2);
pthread_mutex_init(&ctx->lock, NULL);
endpoint_t ep;
make_endpoint(&ep);
/* Build a real-vote endpoint context as if raft_cb_send_vote_request had
* already connected it (on_vote_connected fired, request sent) -- then
* the connection fails before any reply arrives: iomgr invokes the
* registered err_handler on this same ep. */
vote_ep_ctx_build_for_test(&ep, &ha, ctx, /*peer_idx=*/0,
4ULL, 100ULL, 50ULL, "ctrl-a");
int rc = vote_ep_err_handler_for_test(&ep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_reply_hook_calls,
"post-connect vote failure must reach vote_reply_dispatch's tally "
"(xce_vote_reply_hook), not the legacy handle_request_vote_reply path "
"which never invokes this hook");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_reply_granted,
"a post-connect failure must be counted as a denied vote (granted=0)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(4ULL, g_reply_term,
"the synthesized denial must carry the candidate's own term");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_reply_peer_idx,
"peer_idx must match the failed endpoint's vote_ep_ctx_t peer_idx");
free(ep.data);
free_ctx_for_vote(ctx);
ha_fini(&ha);
printf("test_post_connect_vote_failure_routes_through_production_tally: PASS\n");
}
/* ── raft-adapter-conformance task 3.2: post-connect PreVote failure routing ──
*
* pre_vote_ep_err_handler's election-phase branch (task 3.1) must count a
* post-connect PreVote endpoint failure through pre_vote_reply_dispatch --
* the SAME production tally path a connect-time failure or a delivered
* PreVote reply uses -- not a legacy no-op path. Requirement 4.3 (mirror of
* 4.2 for the PreVote endpoint kind).
*
* Test seam: pre_vote_reply_dispatch has no mock hook analogous to
* xce_vote_reply_hook (it calls raft_on_pre_vote_reply directly). Instead we
* assert directly on raft_state_t's own PreVote tally fields
* (pre_vote_replies[], pre_votes_granted, pre_vote_in_progress) -- the same
* fields test_ha_prevote.c already asserts on for the delivered-reply path
* (pre_vote_reply_dispatch_for_test). Pre-fix, pre_vote_ep_err_handler routed
* this failure through the legacy handle_request_vote_reply path, which never
* touches raft_on_pre_vote_reply or these fields -- so pre_vote_replies[1]
* would stay 0 and this test would fail against pre-fix code.
*/
extern int pre_vote_ep_err_handler_for_test(endpoint_t *ep);
extern void pre_vote_reply_dispatch_for_test(ha_state_t *ha, cluster_t *c,
int peer_idx, int granted);
/* A LOST round re-arms the liveness watchdog (ha_prevote_rearm_liveness).
* With the hook unset that falls through to the real
* ha_arm_heartbeat_timeout_vtask, which -- with c==NULL in unit-test mode --
* calloc's a ctx into the static s_liveness_ctx and never arms a delaymgr
* task to free it (delay_task_ms is gated on `c && c->delaymgr.running`),
* leaking it and leaving a stale pointer into this test's stack-local `ha`
* once the test returns. test_ha_prevote.c avoids this identical hazard on
* its own LOST-path tests via the same hook override -- mirrored here. */
extern int (*ha_prevote_rearm_liveness_hook)(ha_state_t *ha, cluster_t *c);
static int stub_prevote_rearm_liveness(ha_state_t *ha, cluster_t *c)
{
(void)ha; (void)c;
return 0;
}
/* Build a 3-node (self + 2 peers) PreVote round in progress, mirroring
* test_ha_prevote.c's make_three_node + ha_start_prevote_async(&ha, NULL)
* fixture: role stays CTRL_STANDBY (the real production role throughout a
* PreVote round -- a node only becomes CTRL_ELECTING after WINNING the
* round), self-grant is recorded by the real raft_begin_pre_vote, and both
* non-self peers (absolute index 1 and 2) start unreplied. */
static void start_three_node_prevote_round(ha_state_t *ha)
{
ha_do_demote(ha);
ha->raft.n_peers = 3;
ha->raft.self_idx = ha->my_priority; /* my_priority == 0 for this fixture */
ha->raft.role = RAFT_FOLLOWER;
ha->n_peers = 3;
ha_start_prevote_async(ha, NULL);
}
static void test_post_connect_prevote_failure_routes_through_production_tally(void)
{
ha_state_t ha;
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &g_vote_cfg, 0));
atomic_store_explicit(&ha.term, 4ULL, memory_order_release);
start_three_node_prevote_round(&ha);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress,
"round must be in progress before the failure arrives");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_votes_granted,
"self-grant must already be recorded by raft_begin_pre_vote");
ha_elect_ctx_t *ctx = alloc_ctx_for_vote(2);
pthread_mutex_init(&ctx->lock, NULL);
endpoint_t ep;
make_endpoint(&ep);
/* Build a PreVote endpoint context as if raft_cb_send_pre_vote_request had
* already connected it (on_vote_connected fired, request sent) -- then
* the connection fails before any reply arrives: iomgr invokes the
* registered err_handler (pre_vote_ep_err_handler) on this same ep. */
vote_ep_ctx_build_for_test(&ep, &ha, ctx, /*peer_idx=*/1,
4ULL, 100ULL, 50ULL, "ctrl-a");
int rc = pre_vote_ep_err_handler_for_test(&ep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_replies[1],
"post-connect PreVote failure must reach pre_vote_reply_dispatch's "
"tally (raft_on_pre_vote_reply), not a legacy no-op path");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_votes_granted,
"a post-connect PreVote failure must be counted as NOT granted "
"(pre_votes_granted stays at the self-grant only)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress,
"round stays pending -- only 1 of 2 non-self peers has been counted, "
"no quorum(2) or all-replied(2) condition met yet");
free(ep.data);
free_ctx_for_vote(ctx);
ha_fini(&ha);
printf("test_post_connect_prevote_failure_routes_through_production_tally: PASS\n");
}
/* ── raft-adapter-conformance task 3.2: fast quorum decision (Req 4.5) ────────
*
* Once every peer in a PreVote round has had its outcome counted -- whether
* via a delivered reply or via this post-connect failure mode -- the round
* must reach its quorum decision immediately (synchronously inside the
* counting call), without waiting for the round's full timeout
* (ha_prevote_round_timeout_fn is never invoked in this test). Mixes both
* counting paths in the same round to prove the failure path is a full
* first-class participant in the fast "all replied" decision, not merely
* counted without unblocking the round.
*/
static void test_prevote_round_reaches_quorum_decision_via_fast_tally(void)
{
ha_state_t ha;
TEST_ASSERT_EQUAL_INT(0, ha_init(&ha, &g_vote_cfg, 0));
atomic_store_explicit(&ha.term, 4ULL, memory_order_release);
start_three_node_prevote_round(&ha);
/* This test's mixed round decides LOST (below), which re-arms the
* liveness watchdog -- stub it out so the real (c==NULL) production path
* is never exercised here (see extern decl comment above). */
ha_prevote_rearm_liveness_hook = stub_prevote_rearm_liveness;
ha_elect_ctx_t *ctx = alloc_ctx_for_vote(2);
pthread_mutex_init(&ctx->lock, NULL);
/* Peer 1: post-connect endpoint failure -- counted via
* pre_vote_ep_err_handler (the fix under test). */
endpoint_t ep1;
make_endpoint(&ep1);
vote_ep_ctx_build_for_test(&ep1, &ha, ctx, /*peer_idx=*/1,
4ULL, 100ULL, 50ULL, "ctrl-a");
TEST_ASSERT_EQUAL_INT(0, pre_vote_ep_err_handler_for_test(&ep1));
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_vote_in_progress,
"round must still be pending -- peer 2's outcome is not counted yet");
/* Peer 2: a normal delivered denial through the production reply-dispatch
* path -- the same tally the post-connect failure above also reaches. */
pre_vote_reply_dispatch_for_test(&ha, NULL, /*peer_idx=*/2, /*granted=*/0);
/* Every non-self peer's outcome is now counted (one via the post-connect
* failure path, one via a delivered reply) -- the round must reach its
* LOST decision immediately, without any call into
* ha_prevote_round_timeout_fn (Requirement 4.5's fast "all replied"
* path). */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, ha.raft.pre_vote_in_progress,
"Req 4.5: once every peer outcome is counted -- including a "
"post-connect failure -- the round must reach its quorum decision "
"without waiting for the round's full timeout");
TEST_ASSERT_EQUAL_INT_MESSAGE(1, ha.raft.pre_votes_granted,
"neither non-self peer granted -- self-grant only, quorum(2) never "
"reached");
TEST_ASSERT_EQUAL_INT_MESSAGE(CTRL_STANDBY,
(int)atomic_load_explicit(&ha.role, memory_order_acquire),
"a LOST PreVote round leaves the node CTRL_STANDBY (Req 1.4, "
"unchanged by this fix)");
ha_prevote_rearm_liveness_hook = NULL; /* reset -- do not leak into other tests */
free(ep1.data);
free_ctx_for_vote(ctx);
ha_fini(&ha);
printf("test_prevote_round_reaches_quorum_decision_via_fast_tally: PASS\n");
}
/* ── Suite entry point ───────────────────────────────────────────────────── */
void test_ha_vote_send(void)
@ -670,4 +900,9 @@ void test_ha_vote_send(void)
RUN_TEST(test_vote_reply_handler_no_uaf);
/* bug-1417 */
RUN_TEST(test_bug1417_middle_index_real_vote_from_peer2_not_dropped);
/* raft-adapter-conformance task 3.1 */
RUN_TEST(test_post_connect_vote_failure_routes_through_production_tally);
/* raft-adapter-conformance task 3.2 */
RUN_TEST(test_post_connect_prevote_failure_routes_through_production_tally);
RUN_TEST(test_prevote_round_reaches_quorum_decision_via_fast_tally);
}

973
src/controller/testsuite/test_persist_snapshot_entities.c
View file

File diff suppressed because it is too large Load diff

131
src/controller/testsuite/test_repl_unit.c
View file

@ -583,6 +583,134 @@ static void test_repl_registration_lifecycle(void) {
printf("test_repl_registration_lifecycle: PASS\n");
}
/* ============================================================
* bug-1474: repl_add_standby slot allocation must never clobber an
* unexpired bug-1147 disconnect guard (last_disconnected_lsn) belonging to
* a DIFFERENT, still-disconnected peer, when a genuinely virgin
* (never-used) slot is available. See .wolf/buglog.json bug-1474 for the
* full forensic trace: a spurious/duplicate reconnect for an unrelated,
* already-connected peer picked the lower-indexed, guard-holding slot over
* a higher-indexed virgin slot and memset it to zero, silently destroying
* the guard repl_trim_lsn relies on to protect WAL segments a genuinely
* disconnected-but-reconnecting standby still needs.
* ============================================================ */
/* RED (pre-fix): repl_add_standby's old "first empty slot by lowest index"
* scan picks idx_a (guard-holding, index 0) over the virgin slot (index 2)
* purely because of index order, destroying A's guard. Req: the guard must
* survive and the new peer must land in the virgin slot instead. */
static void test_repl_add_standby_prefers_virgin_slot_over_guard(void) {
replicator_t r;
int ep_a = 0, ep_b = 1, ep_c = 2;
repl_init(&r);
int idx_a = repl_add_standby(&r, (void *)&ep_a, "standby-a", 7001);
int idx_b = repl_add_standby(&r, (void *)&ep_b, "standby-b", 7002);
TEST_ASSERT_TRUE(idx_a >= 0);
TEST_ASSERT_TRUE(idx_b >= 0);
TEST_ASSERT_NOT_EQUAL(idx_a, idx_b);
/* A acks up to LSN 42, then disconnects: leaves a bug-1147 guard at
* idx_a with last_disconnected_lsn == 42. The third slot (whichever
* index neither idx_a nor idx_b used) remains genuinely virgin. */
repl_on_ack(&r, (void *)&ep_a, 42);
repl_remove_standby(&r, (void *)&ep_a);
TEST_ASSERT_NULL(r.standbys[idx_a].ep);
TEST_ASSERT_EQUAL_UINT64(42, r.standbys[idx_a].last_disconnected_lsn);
/* A third, genuinely NEW peer (unrelated to A) connects while idx_a's
* guard is still unexpired and a virgin slot is also available. */
int idx_c = repl_add_standby(&r, (void *)&ep_c, "standby-c", 7003);
TEST_ASSERT_TRUE(idx_c >= 0);
/* The new peer must land in the virgin slot, NOT the guard-holding one,
* even though idx_a may be the lower index. */
TEST_ASSERT_NOT_EQUAL_MESSAGE(idx_a, idx_c,
"new peer must not be allocated the guard-holding slot when a"
" virgin slot exists (bug-1474)");
/* Load-bearing assertion: A's guard must survive completely untouched.
* Before the fix, repl_add_standby's unconditional memset(s, 0, ...)
* on the chosen slot would have wiped last_disconnected_lsn to 0 here. */
TEST_ASSERT_NULL_MESSAGE(r.standbys[idx_a].ep,
"guard-holding slot must remain disconnected (bug-1474)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(42, r.standbys[idx_a].last_disconnected_lsn,
"bug-1147 guard LSN must survive an unrelated peer's"
" repl_add_standby call (bug-1474)");
repl_fini(&r);
printf("test_repl_add_standby_prefers_virgin_slot_over_guard: PASS\n");
}
/* RED (pre-fix, and proves the DELIBERATE fallback policy): with
* MAX_STANDBYS==3, exhaust all three slots, then disconnect two of them
* (both now guard-holding, one older-disconnected than the other) so NO
* virgin slot remains. A fourth genuinely new peer must still be admitted
* (never -ENOSPC for this case -- refusing a legitimate reconnect would be
* worse for cluster availability than an early guard loss), and the
* fallback must deterministically reuse the OLDER-disconnected guard slot
* (closest to its own natural repl_expire_disconnect_slots expiry, so
* reusing it discards the LEAST remaining WAL-retention protection) --
* not merely the lowest-indexed guard slot. */
static void test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect(void) {
replicator_t r;
int ep_a = 0, ep_b = 1, ep_c = 2, ep_d = 3;
repl_init(&r);
int idx_a = repl_add_standby(&r, (void *)&ep_a, "standby-a", 7001);
int idx_b = repl_add_standby(&r, (void *)&ep_b, "standby-b", 7002);
int idx_c = repl_add_standby(&r, (void *)&ep_c, "standby-c", 7003);
TEST_ASSERT_TRUE(idx_a >= 0);
TEST_ASSERT_TRUE(idx_b >= 0);
TEST_ASSERT_TRUE(idx_c >= 0);
/* All MAX_STANDBYS(3) slots now occupied. */
/* B disconnects first; force its disconnect timestamp strictly OLDER
* than A's (below) so the two guards are unambiguously ordered by
* time, independent of their slot index (idx_a < idx_b by add order --
* deliberately the OPPOSITE of the time order, so a naive
* "lowest-index guard slot" fallback would pick the wrong one). */
repl_on_ack(&r, (void *)&ep_b, 20);
repl_remove_standby(&r, (void *)&ep_b);
r.standbys[idx_b].last_disconnect_time -= 100;
/* A disconnects second, with its real (newer) "now" timestamp. */
repl_on_ack(&r, (void *)&ep_a, 10);
repl_remove_standby(&r, (void *)&ep_a);
TEST_ASSERT_TRUE_MESSAGE(
r.standbys[idx_b].last_disconnect_time < r.standbys[idx_a].last_disconnect_time,
"test setup invariant: B must be the OLDER disconnect");
/* C is still connected; nstandbys == 1 (< MAX_STANDBYS), so the
* "too many distinct live peers" -ENOSPC guard does not trip -- but
* BOTH empty slots (idx_a, idx_b) are guard-holding: no virgin slot
* exists anywhere in the registry. */
TEST_ASSERT_EQUAL_INT(1, r.nstandbys);
/* A fourth, genuinely new peer connects into the exhausted state. */
int idx_d = repl_add_standby(&r, (void *)&ep_d, "standby-d", 7004);
TEST_ASSERT_TRUE_MESSAGE(idx_d >= 0,
"exhausted-guard fallback must still admit a legitimate new peer,"
" never -ENOSPC (bug-1474 policy)");
/* Deliberate policy: reuse the OLDER-disconnected guard slot (idx_b),
* not idx_a (which is both newer-disconnected AND lower-indexed --
* ruling out an accidental "first guard by index" fallthrough). */
TEST_ASSERT_EQUAL_INT_MESSAGE(idx_b, idx_d,
"exhausted fallback must reuse the OLDEST-disconnected guard slot,"
" not merely the lowest-indexed one (bug-1474 policy)");
/* idx_a's guard (the one NOT chosen) must remain completely untouched. */
TEST_ASSERT_NULL_MESSAGE(r.standbys[idx_a].ep,
"non-selected guard slot must remain disconnected");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(10, r.standbys[idx_a].last_disconnected_lsn,
"non-selected guard slot's LSN must survive untouched");
repl_fini(&r);
printf("test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect: PASS\n");
}
void test_repl_unit(void) {
UnitySetTestFile(__FILE__);
RUN_TEST(test_repl_init_zeroes_nstandbys);
@ -608,4 +736,7 @@ void test_repl_unit(void) {
RUN_TEST(test_wal_nack_does_not_update_acked_lsn);
/* Task 5.3: registration lifecycle — Req 1.1, 1.2, 1.3 */
RUN_TEST(test_repl_registration_lifecycle);
/* bug-1474: slot allocation must not clobber an unexpired guard */
RUN_TEST(test_repl_add_standby_prefers_virgin_slot_over_guard);
RUN_TEST(test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect);
}

272
src/controller/testsuite/test_snapverify.c
View file

@ -26,15 +26,41 @@ static void cleanup_dir(void)
}
/* Writes a checkpoint file matching persist.c write_snapshot()'s exact
* on-disk layout (magic+lsn+cfgref(41, all-zero/absent) + fmt + job_count +
* per-job[len+buf] + queue_count + per-queue[namelen+name+status]).
* job_lens/job_count and queue_names/queue_count are caller-controlled so
* tests can inject a specific defect. `truncate_last_bytes` drops that many
* bytes off the END of the fully-written file (0 = write it whole). */
static void write_snap(const char* path, uint64_t lsn, uint32_t fmt,
const uint32_t* job_lens, uint32_t job_count,
const char** queue_names, const uint32_t* queue_status,
uint32_t queue_count, int truncate_last_bytes)
* on-disk layout for SNAPSHOT_FORMAT_VERSION=2 (magic+lsn+cfgref(41) + fmt +
* job_count + per-job[payload_len+buf] + queue_count + per-queue
* [payload_len+opaque_bytes] + host_count + per-host[payload_len+buf] +
* cfg_bytes_len + cfg_bytes).
*
* Queue/host frame CONTENT is irrelevant here -- snapverify validates
* structure only (length-prefix walk), never capnp-decodes a payload -- so
* queue frames are written as an opaque [payload_len][name_len+name+status]
* blob (convenient filler reusing the queue_names/queue_status test
* parameters, NOT a real capnp QueueAdminState encoding) and host frames as
* an opaque [payload_len][0xCD-filled bytes] blob.
*
* job_lens/job_count, queue_names/queue_status/queue_count, and
* host_lens/host_count are all caller-controlled so tests can inject a
* specific defect or size at a specific section. cfg_bytes_len controls the
* trailing config-bytes section's declared length (content is 0xEF filler);
* pass 0 for "section present but empty" -- the section's 4-byte length
* field is ALWAYS written by this helper (mirrors persist.c writing it on
* every checkpoint, even when there are no local config bytes yet). Tests
* that need to inject a real header cfgref, or a file with NO config-bytes
* length field at all, use write_snap_full()/a hand-written file instead
* (see below).
*
* `truncate_last_bytes` drops that many bytes off the END of the fully-
* written file (0 = write it whole). */
static void write_snap_full(const char* path, uint64_t lsn,
int cfgref_present, uint64_t cfgref_tag,
const uint8_t* cfgref_sha /* 32 bytes, or NULL
* for all-zero */,
uint32_t fmt,
const uint32_t* job_lens, uint32_t job_count,
const char** queue_names,
const uint32_t* queue_status, uint32_t queue_count,
const uint32_t* host_lens, uint32_t host_count,
uint32_t cfg_bytes_len, int truncate_last_bytes)
{
FILE* fp = fopen(path, "wb");
TEST_ASSERT_NOT_NULL(fp);
@ -43,6 +69,10 @@ static void write_snap(const char* path, uint64_t lsn, uint32_t fmt,
memset(header, 0, sizeof(header));
memcpy(header, "XCESNAP\0", 8);
memcpy(header + 8, &lsn, 8);
header[16] = (char)(cfgref_present ? 1 : 0);
memcpy(header + 17, &cfgref_tag, 8);
if (cfgref_sha)
memcpy(header + 25, cfgref_sha, 32);
fwrite(header, 1, sizeof(header), fp);
fwrite(&fmt, sizeof(fmt), 1, fp);
@ -57,11 +87,31 @@ static void write_snap(const char* path, uint64_t lsn, uint32_t fmt,
fwrite(&queue_count, sizeof(queue_count), 1, fp);
for (uint32_t i = 0; i < queue_count; i++) {
uint32_t nlen = (uint32_t)strlen(queue_names[i]);
uint32_t nlen = (uint32_t)strlen(queue_names[i]);
uint32_t payload_len = (uint32_t)(sizeof(nlen) + nlen +
sizeof(queue_status[i]));
fwrite(&payload_len, sizeof(payload_len), 1, fp);
fwrite(&nlen, sizeof(nlen), 1, fp);
fwrite(queue_names[i], 1, nlen, fp);
fwrite(&queue_status[i], sizeof(queue_status[i]), 1, fp);
}
fwrite(&host_count, sizeof(host_count), 1, fp);
for (uint32_t i = 0; i < host_count; i++) {
fwrite(&host_lens[i], sizeof(host_lens[i]), 1, fp);
uint8_t* buf = malloc(host_lens[i]);
memset(buf, 0xCD, host_lens[i]);
fwrite(buf, 1, host_lens[i], fp);
free(buf);
}
fwrite(&cfg_bytes_len, sizeof(cfg_bytes_len), 1, fp);
if (cfg_bytes_len > 0) {
uint8_t* buf = malloc(cfg_bytes_len);
memset(buf, 0xEF, cfg_bytes_len);
fwrite(buf, 1, cfg_bytes_len, fp);
free(buf);
}
fclose(fp);
if (truncate_last_bytes > 0) {
@ -74,6 +124,21 @@ static void write_snap(const char* path, uint64_t lsn, uint32_t fmt,
}
}
/* Convenience wrapper for the (common) case of no header cfgref -- mirrors
* every pre-task-8.4 call site's shape, just with the new host_lens/
* host_count/cfg_bytes_len parameters appended. */
static void write_snap(const char* path, uint64_t lsn, uint32_t fmt,
const uint32_t* job_lens, uint32_t job_count,
const char** queue_names, const uint32_t* queue_status,
uint32_t queue_count,
const uint32_t* host_lens, uint32_t host_count,
uint32_t cfg_bytes_len, int truncate_last_bytes)
{
write_snap_full(path, lsn, 0, 0, NULL, fmt, job_lens, job_count,
queue_names, queue_status, queue_count, host_lens,
host_count, cfg_bytes_len, truncate_last_bytes);
}
static void test_legacy_position_only_file_is_ok(void)
{
fresh_dir();
@ -104,21 +169,29 @@ static void test_entity_bearing_well_formed_file_is_ok(void)
char path[512];
snprintf(path, sizeof(path), "%s/snap-0000000000000064.capnp", g_dir);
uint32_t job_lens[3] = { 100, 200, 50 };
uint32_t job_lens[3] = { 100, 200, 50 };
const char* qnames[2] = { "gpu", "cpu" };
uint32_t qstatus[2] = { 1, 0 };
write_snap(path, 100, 1, job_lens, 3, qnames, qstatus, 2, 0);
uint32_t qstatus[2] = { 1, 0 };
/* fmt=2 (SNAPSHOT_FORMAT_VERSION), no hosts, cfg-bytes section present
* but empty -- must still be fully consumed to EOF by the fmt-2-aware
* parser (task 8.4 fix: this file shape is what a real checkpoint
* written by the current codebase looks like). */
write_snap(path, 100, 2, job_lens, 3, qnames, qstatus, 2, NULL, 0, 0, 0);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.ok);
TEST_ASSERT_EQUAL_INT(1, rep.has_entity_section);
TEST_ASSERT_EQUAL_UINT32(1, rep.format_version);
TEST_ASSERT_EQUAL_UINT32(2, rep.format_version);
TEST_ASSERT_EQUAL_UINT32(3, rep.job_count);
TEST_ASSERT_EQUAL_UINT32(3, rep.jobs_ok);
TEST_ASSERT_EQUAL_UINT32(2, rep.queue_count);
TEST_ASSERT_EQUAL_UINT32(2, rep.queues_ok);
TEST_ASSERT_EQUAL_UINT32(0, rep.host_count);
TEST_ASSERT_EQUAL_UINT32(0, rep.hosts_ok);
TEST_ASSERT_EQUAL_INT(1, rep.has_config_bytes_section);
TEST_ASSERT_EQUAL_UINT32(0, rep.config_bytes_len);
TEST_ASSERT_EQUAL_INT(-1, (int)rep.fail_offset);
cleanup_dir();
}
@ -165,7 +238,8 @@ static void test_unrecognized_format_version_is_einval(void)
char path[512];
snprintf(path, sizeof(path), "%s/snap-0000000000000001.capnp", g_dir);
uint32_t job_lens[1] = { 10 };
write_snap(path, 1, 99 /* bogus version */, job_lens, 1, NULL, NULL, 0, 0);
write_snap(path, 1, 99 /* bogus version */, job_lens, 1, NULL, NULL, 0,
NULL, 0, 0, 0);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
@ -187,7 +261,7 @@ static void test_corrupt_job_payload_len_reports_exact_failure_point(void)
snprintf(path, sizeof(path), "%s/snap-0000000000000002.capnp", g_dir);
uint32_t job_lens[4] = { 50, 60, 70, 80 };
write_snap(path, 2, 1, job_lens, 4, NULL, NULL, 0, 0);
write_snap(path, 2, 2, job_lens, 4, NULL, NULL, 0, NULL, 0, 0, 0);
/* Corrupt job 2's length field (3rd job, after header(57)+fmt(4)+
* job_count(4)+job0(4+50)+job1(4+60) = 57+4+4+54+64 = 183) to a value
@ -217,6 +291,166 @@ static void test_missing_path_errors(void)
TEST_ASSERT_LESS_THAN_INT(0, rc);
}
/* task 8.4: the header's cfgref tag/sha must be DECODED (not merely
* reported present) -- a genuine round-trip assertion against concrete
* injected values, not just "non-zero". */
static void test_cfgref_tag_and_sha_decode_roundtrip(void)
{
fresh_dir();
char path[512];
snprintf(path, sizeof(path), "%s/snap-00000000000000c8.capnp", g_dir);
uint64_t tag = 0x1122334455667788ULL;
uint8_t sha[32];
memset(sha, 0xAA, 16);
memset(sha + 16, 0xBB, 16);
write_snap_full(path, 200, 1 /* cfgref_present */, tag, sha, 2, NULL, 0,
NULL, NULL, 0, NULL, 0, 0, 0);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.ok);
TEST_ASSERT_EQUAL_INT(1, rep.has_cfgref);
TEST_ASSERT_EQUAL_UINT64(tag, rep.cfgref_tag);
TEST_ASSERT_EQUAL_MEMORY(sha, rep.cfgref_sha, 32);
cleanup_dir();
}
/* task 8.4: host frames get their own structural walk + counts, same shape
* as the job/queue loops. */
static void test_host_frames_are_walked_and_counted(void)
{
fresh_dir();
char path[512];
snprintf(path, sizeof(path), "%s/snap-00000000000000c9.capnp", g_dir);
uint32_t host_lens[3] = { 30, 45, 12 };
write_snap(path, 201, 2, NULL, 0, NULL, NULL, 0, host_lens, 3, 0, 0);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.ok);
TEST_ASSERT_EQUAL_UINT32(3, rep.host_count);
TEST_ASSERT_EQUAL_UINT32(3, rep.hosts_ok);
cleanup_dir();
}
/* task 8.4: a non-empty trailing config-bytes section is reported with its
* concrete declared length. */
static void test_nonempty_config_bytes_section_is_reported(void)
{
fresh_dir();
char path[512];
snprintf(path, sizeof(path), "%s/snap-00000000000000ca.capnp", g_dir);
write_snap(path, 202, 2, NULL, 0, NULL, NULL, 0, NULL, 0, 42, 0);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.ok);
TEST_ASSERT_EQUAL_INT(1, rep.has_config_bytes_section);
TEST_ASSERT_EQUAL_UINT32(42, rep.config_bytes_len);
cleanup_dir();
}
/* task 8.4: an EXPLICIT zero-length config-bytes section (the 4-byte length
* field IS present and reads 0, immediately followed by EOF) must be
* distinguishable from a file that has no config-bytes section at all --
* both are ok==1, but only the explicit-zero case has
* has_config_bytes_section==1. */
static void test_explicit_zero_length_config_bytes_section_is_ok(void)
{
fresh_dir();
char path[512];
snprintf(path, sizeof(path), "%s/snap-00000000000000cb.capnp", g_dir);
write_snap(path, 203, 2, NULL, 0, NULL, NULL, 0, NULL, 0, 0, 0);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.ok);
TEST_ASSERT_EQUAL_INT(1, rep.has_config_bytes_section);
TEST_ASSERT_EQUAL_UINT32(0, rep.config_bytes_len);
cleanup_dir();
}
/* task 8.4: a file that simply ENDS right after the host section (no
* config_bytes_len field at all -- a slightly-older-but-still-fmt-2-
* consistent shape) is still ok==1, but has_config_bytes_section==0 --
* the counterpart to the explicit-zero-length case above, proving the two
* are genuinely distinguishable rather than collapsed into one "empty"
* bucket. Hand-written (not via write_snap_full, which always writes the
* length field) to produce exactly this shape. */
static void test_no_config_bytes_section_at_all_is_still_ok(void)
{
fresh_dir();
char path[512];
snprintf(path, sizeof(path), "%s/snap-00000000000000cc.capnp", g_dir);
FILE* fp = fopen(path, "wb");
TEST_ASSERT_NOT_NULL(fp);
char header[16 + 1 + 8 + 32];
memset(header, 0, sizeof(header));
memcpy(header, "XCESNAP\0", 8);
uint64_t lsn = 204;
memcpy(header + 8, &lsn, 8);
fwrite(header, 1, sizeof(header), fp);
uint32_t fmt = 2;
uint32_t zero = 0;
fwrite(&fmt, sizeof(fmt), 1, fp); /* format_version */
fwrite(&zero, sizeof(zero), 1, fp); /* job_count = 0 */
fwrite(&zero, sizeof(zero), 1, fp); /* queue_count = 0 */
fwrite(&zero, sizeof(zero), 1, fp); /* host_count = 0 */
/* No config_bytes_len field at all -- file ends right here. */
fclose(fp);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_INT(1, rep.ok);
TEST_ASSERT_EQUAL_INT(0, rep.has_config_bytes_section);
cleanup_dir();
}
/* task 8.4: same corrupt-length-field diagnostic as
* test_corrupt_job_payload_len_reports_exact_failure_point, in the host
* section instead of the job section. */
static void test_corrupt_host_payload_len_reports_exact_failure_point(void)
{
fresh_dir();
char path[512];
snprintf(path, sizeof(path), "%s/snap-00000000000000cd.capnp", g_dir);
uint32_t host_lens[3] = { 20, 30, 40 };
write_snap(path, 205, 2, NULL, 0, NULL, NULL, 0, host_lens, 3, 0, 0);
/* header(57) + fmt(4) + job_count(4) + [0 jobs] + queue_count(4) +
* [0 queues] + host_count(4) + host0(4+20) + host1(4+30) = offset of
* host2's payload_len field. */
long host2_len_off = 57 + 4 + 4 + 4 + 4 + (4 + 20) + (4 + 30);
FILE* fp = fopen(path, "r+b");
TEST_ASSERT_NOT_NULL(fp);
fseek(fp, host2_len_off, SEEK_SET);
uint32_t garbage = 0x7FFFFFFF;
fwrite(&garbage, sizeof(garbage), 1, fp);
fclose(fp);
snapverify_report_t rep;
int rc = snapverify_path(path, stdout, &rep);
TEST_ASSERT_EQUAL_INT(-EIO, rc);
TEST_ASSERT_EQUAL_INT(0, rep.ok);
TEST_ASSERT_EQUAL_UINT32(2, rep.hosts_ok); /* host 0, host 1 parsed fine */
TEST_ASSERT_EQUAL_INT(host2_len_off, (int)rep.fail_offset);
cleanup_dir();
}
void test_snapverify(void)
{
UnitySetTestFile(__FILE__);
@ -227,4 +461,10 @@ void test_snapverify(void)
RUN_TEST(test_unrecognized_format_version_is_einval);
RUN_TEST(test_corrupt_job_payload_len_reports_exact_failure_point);
RUN_TEST(test_missing_path_errors);
RUN_TEST(test_cfgref_tag_and_sha_decode_roundtrip);
RUN_TEST(test_host_frames_are_walked_and_counted);
RUN_TEST(test_nonempty_config_bytes_section_is_reported);
RUN_TEST(test_explicit_zero_length_config_bytes_section_is_ok);
RUN_TEST(test_no_config_bytes_section_at_all_is_still_ok);
RUN_TEST(test_corrupt_host_payload_len_reports_exact_failure_point);
}

180
src/controller/testsuite/test_term_safety_adapter.c
View file

@ -1983,6 +1983,181 @@ static void test_repl_ack_out_of_range_peer_rejected(void)
printf("test_repl_ack_out_of_range_peer_rejected: PASS\n");
}
/* ================================================================== */
/* bug-1479 — drive the library on a REJECTED AppendEntries (WAL NACK) */
/* ================================================================== */
/* Before this fix, a WAL NACK (WalAckReply.success == 0) for an ordinary
* Raft log conflict (ae_reject_logmismatch) was silently dropped after
* stale-term discovery -- the library's own, already-correct nextIndex
* backoff (raft_on_append_entries_reply's failure branch, raft.tla 401) was
* never invoked, so a standby stuck in log conflict never had its resend
* position corrected: the primary kept broadcasting new entries assuming
* agreement at an index the standby had already rejected, forever, until
* MAX_PENDING_ACKS silently evicted it into catch-up -- at which point
* NOTHING streamed it back into sync either (the ACK-driven continuation
* needs a real ACK to fire, which a log-conflicted standby can never send).
*
* ha_raft_on_repl_nack mirrors ha_raft_on_repl_ack's bounds-checked
* peer-index resolution, but routes success=0 into the library (backing
* off next_index[peer] by one, per raft.c's documented failure branch) and
* returns the corrected resume LSN instead of trying to advance commit. */
/* (a) A rejection backs off next_index[peer] by exactly one and returns the
* corrected LSN (next_index - 1) -- the position the standby is now
* believed caught up through. */
static void test_repl_nack_backs_off_next_index_and_returns_corrected_lsn(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_backoff_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
/* make_leader pokes role=RAFT_LEADER directly without going through the
* library's real BecomeLeader transition (raft_init_leader_progress,
* raft.c:97-103), so next_index[] stays 0 for every peer regardless of
* log_len. Seed it here to what a genuine BecomeLeader would have set
* (log_len + 1), so this test exercises a realistic starting position. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
TEST_ASSERT_EQUAL_UINT64(6, c->ha.raft.next_index[1]);
uint64_t corrected_lsn = 999; /* sentinel -- must be overwritten */
int rc = ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn);
TEST_ASSERT_EQUAL_INT(0, rc);
/* next_index backed off by exactly one (raft.c: Max(next_index-1, 1)). */
TEST_ASSERT_EQUAL_UINT64(5, c->ha.raft.next_index[1]);
/* Corrected resume point is next_index - 1. */
TEST_ASSERT_EQUAL_UINT64(4, corrected_lsn);
/* A rejection never advances commit. */
TEST_ASSERT_EQUAL_UINT64(0, c->ha.raft.commit_index);
teardown_cluster(c);
printf("test_repl_nack_backs_off_next_index_and_returns_corrected_lsn: PASS\n");
}
/* (b) Repeated rejections keep backing off, floored at 1 -- never underflows
* (raft.c's Max(next_index-1, 1), matching raft.tla exactly). */
static void test_repl_nack_repeated_rejections_floor_at_one(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_floor_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] to what a real BecomeLeader would set (log_len+1 =
* 3) -- make_leader itself leaves it at 0 (see the backoff test above).
* Reject enough times to walk it down to 1 and confirm it never goes
* below. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
for (int i = 0; i < 5; i++) {
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T,
&corrected_lsn));
}
TEST_ASSERT_EQUAL_UINT64(1, c->ha.raft.next_index[1]);
TEST_ASSERT_EQUAL_UINT64(0, corrected_lsn);
teardown_cluster(c);
printf("test_repl_nack_repeated_rejections_floor_at_one: PASS\n");
}
/* (c) Defense-in-depth: same out-of-range/self rejection as the ACK seam --
* no library accounting on a bogus peer index. */
static void test_repl_nack_out_of_range_peer_rejected(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_oob_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/2);
/* Seed next_index[1] as a real BecomeLeader would (log_len+1 = 3); see
* the backoff test above for why make_leader itself leaves it at 0. */
c->ha.raft.next_index[1] = c->ha.raft.log_len + 1;
uint64_t corrected_lsn = 0;
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/3, /*reply_term*/T, &corrected_lsn));
TEST_ASSERT_EQUAL_INT(-EINVAL,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/0, /*reply_term*/T, &corrected_lsn));
/* next_index untouched by either rejected call. */
TEST_ASSERT_EQUAL_UINT64(3, c->ha.raft.next_index[1]);
teardown_cluster(c);
printf("test_repl_nack_out_of_range_peer_rejected: PASS\n");
}
/* (d) THE CORE REGRESSION: composing ha_raft_on_repl_nack + repl_start_catchup
* -- exactly what repl_ack_decode_and_dispatch's NACK branch now does --
* marks the standby in_catch_up at the CORRECTED position on the very FIRST
* rejection. Before this fix, nothing marked it at all until repl_broadcast's
* own MAX_PENDING_ACKS=16 eviction fired (silently, using the standby's last
* ACKED lsn rather than a library-backed-off position) -- and even then,
* nothing would ever stream to it (the ACK-driven continuation requires a
* real ACK a log-conflicted standby can never send). This test proves the
* immediate half of the fix: no 16-strike wait, and the resume point comes
* from the library's own nextIndex backoff, not a guess. */
static void test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes(void)
{
char tmp[256];
strcpy(tmp, "/tmp/xce_ts_nack_catchup_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(tmp));
const uint64_t T = 6;
cluster_t* c = make_cluster(tmp);
TEST_ASSERT_NOT_NULL(c);
make_leader(c, T, /*n_peers*/3);
leader_seed_entries(c, T, /*count*/5);
TEST_ASSERT_EQUAL_INT(0,
repl_add_standby(&c->persist.repl, &g_ae_dummy_ep, "sb0", 7000));
uint64_t join_lsn_before;
TEST_ASSERT_EQUAL_INT_MESSAGE(0,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_before),
"a freshly-added standby must not already be in catch-up");
/* THE FIRST rejection -- not the 16th. */
uint64_t corrected_lsn = 999;
TEST_ASSERT_EQUAL_INT(0,
ha_raft_on_repl_nack(c, /*abs_peer_idx*/1, /*reply_term*/T, &corrected_lsn));
repl_start_catchup(&c->persist.repl, &g_ae_dummy_ep, corrected_lsn);
uint64_t join_lsn_after;
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
repl_get_catchup_join_lsn(&c->persist.repl, &g_ae_dummy_ep, &join_lsn_after),
"a single rejection must immediately mark the standby in catch-up "
"(bug-1479 -- no 16-strike wait)");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(corrected_lsn, join_lsn_after,
"the catch-up resume point must be the library's backed-off "
"next_index - 1, not a guess");
teardown_cluster(c);
printf("test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes: PASS\n");
}
/* ================================================================== */
/* Task 5.1 — every-callback + marker-path end-to-end integration */
/* ================================================================== */
@ -2209,6 +2384,11 @@ void test_term_safety_adapter(void)
RUN_TEST(test_repl_ack_abs_peer_idx_endpoint_type_roundtrip);
RUN_TEST(test_repl_ack_heartbeat_standby_absolute_slot);
RUN_TEST(test_repl_ack_out_of_range_peer_rejected);
/* bug-1479: NACK seam drives the library's nextIndex backoff. */
RUN_TEST(test_repl_nack_backs_off_next_index_and_returns_corrected_lsn);
RUN_TEST(test_repl_nack_repeated_rejections_floor_at_one);
RUN_TEST(test_repl_nack_out_of_range_peer_rejected);
RUN_TEST(test_nack_immediately_marks_catchup_at_corrected_lsn_not_after_16_strikes);
/* Task 5.1: every-callback + marker-path end-to-end integration. */
RUN_TEST(test_every_callback_and_marker_path_end_to_end);
}

1072
src/controller/testsuite/test_upgrade_snapshot_apply.c Normal file
View file

File diff suppressed because it is too large Load diff

361
src/controller/testsuite/test_upgrade_snapshot_restore.c Normal file
View file

@ -0,0 +1,361 @@
/*
* test_upgrade_snapshot_restore.c — config-upgrade-snapshot task 3
*
* TDD tests for restore_host_state() (Requirements 1.10, 1.11, 5.8):
* restoring a host admin-state frame must work whether or not the target
* host has ever joined this member (create-if-missing, since hosts are
* join-provisioned, not config-provisioned -- Design Invariant 1), must be
* idempotent under a retried restore (Design Invariant 3), must never
* clobber a pre-existing worker's non-admin state on a hit, and the
* placeholder it creates on a miss must be safe for the host's later real
* join (routed through rejoin_worker by join.c's own existing
* key_find-based dispatch, unmodified) to adopt -- admin status set on the
* placeholder must still be set after that real join (Design Invariant 6).
*
* RED phase: restore_host_state does not exist yet, so this file fails to
* compile/link until Task 3 adds it to worker.c + worker.h.
*
* Fixture mirrors test_worker_join.c's bare in-memory cluster_t (workermgr +
* resmgr kvmgr instances, no disk/persist) -- restore_host_state only
* touches c->workermgr directly, and the placeholder-then-real-join test
* additionally needs the resmgr resource defs that apply_join_state's
* add_hostattrs_resvals()/set_host_slots() (invoked via rejoin_worker)
* require -- same resource-name list as test_worker_join.c.
*
* collect_confvars() (required for worker.c to link without pulling in the
* full dataex.c) is intentionally NOT redefined here: test_worker_join.c
* already provides it once for the whole controller_unit_test binary.
*/
#include <rtti.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unity/unity.h>
#include "cluster.h"
#include "kvmgr.h"
#include "resobj.h"
#include "wobj.h"
#include "worker.h"
#include "xce/cluster_def.h"
#include "xce/types.h"
#include "xstring.h"
#include "zarray.h"
/* ── Test fixture ────────────────────────────────────────────────────── */
/* Resource names/types matching add_hostattrs_resvals/set_host_slots in
* worker.c -- required so a real join (rejoin_worker) after a placeholder
* restore succeeds instead of failing resource lookups. */
#define N_RES 9
static const char* g_res_names[N_RES] = {
"type", "model", "cpuf", "maxcpus", "maxmem",
"maxswap", "maxtmp", "ndisks", "slots"
};
static const int g_res_valtypes[N_RES] = {
XCE_VAL_STRING, XCE_VAL_STRING,
XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC,
XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC,
XCE_VAL_NUMERIC
};
typedef struct {
cluster_t c;
var workermgr;
var resmgr;
} tur_cluster_t;
static tur_cluster_t g_tur;
static void tur_setup(void)
{
kvmgr_params_t str_params = {
.key_type = KVMGR_KEY_STRING,
.nslots = 64,
.delta = 16,
.datakiller = NULL,
};
/* Idempotent guard: the xstring intern table is process-global and
* shared across every test file in this binary (mirrors
* test_apply_seam.c / test_replay_apply_seam.c's guard, safer than
* test_worker_join.c's unconditional call since both files run in the
* same controller_unit_test process). */
if (makeString((char*)"tur-probe") == NULL)
xstring_table_init(139);
memset(&g_tur, 0, sizeof(g_tur));
g_tur.workermgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(g_tur.workermgr);
g_tur.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(g_tur.resmgr);
for (int i = 0; i < N_RES; i++) {
xce_resource_t rdef;
memset(&rdef, 0, sizeof(rdef));
rdef.name = (char*)g_res_names[i];
rdef.valtype = g_res_valtypes[i];
rdef.flags = 0;
rdef.idx = -1;
xce_resource_t* r = makeResource(g_tur.resmgr, &rdef);
TEST_ASSERT_NOT_NULL_MESSAGE(r, g_res_names[i]);
}
g_tur.c.workermgr = g_tur.workermgr;
g_tur.c.resmgr = g_tur.resmgr;
zarray_init(&g_tur.c.sharedinst, 64, NULL);
}
static void tur_teardown(void)
{
zarray_fini(&g_tur.c.sharedinst);
if (g_tur.resmgr) dereference(g_tur.resmgr);
if (g_tur.workermgr) dereference(g_tur.workermgr);
memset(&g_tur, 0, sizeof(g_tur));
}
static worker_t* tur_find(const char* name)
{
worker_t* w = NULL;
lock(g_tur.c.workermgr, RTTI_LOCK_READ);
key_find(g_tur.c.workermgr, (void*)name, (void**)&w);
unlock(g_tur.c.workermgr);
return w;
}
/* ── Miss: create-if-missing placeholder (Req 1.10) ────────────────────── */
static void test_restore_host_state_creates_placeholder_for_unknown_host(void)
{
tur_setup();
int rc = restore_host_state(&g_tur.c, "never-joined-host",
XCE_WORKER_CLOSED);
TEST_ASSERT_EQUAL_INT(0, rc);
worker_t* w = tur_find("never-joined-host");
TEST_ASSERT_NOT_NULL_MESSAGE(w,
"restore for a host with no local record must create exactly one placeholder");
TEST_ASSERT_TRUE_MESSAGE(w->host.status & XCE_WORKER_CLOSED,
"restored admin bit (CLOSED) must be set on the placeholder");
TEST_ASSERT_TRUE_MESSAGE(w->host.status & XCE_WORKER_UNAVAIL,
"placeholder must be unjoined/dead-until-connected");
TEST_ASSERT_NOT_NULL_MESSAGE(w->host.attrs,
"placeholder hostattrs must be non-NULL (rejoin_worker "
"dereferences it directly on the later real join)");
tur_teardown();
printf("test_restore_host_state_creates_placeholder_for_unknown_host: PASS\n");
}
/* ── Idempotent retry after partial failure (Req 5.8) ──────────────────── */
static void test_restore_host_state_retry_is_idempotent(void)
{
tur_setup();
int rc1 = restore_host_state(&g_tur.c, "retry-host", XCE_WORKER_CLOSED);
TEST_ASSERT_EQUAL_INT(0, rc1);
worker_t* w1 = tur_find("retry-host");
TEST_ASSERT_NOT_NULL(w1);
int idx1 = w1->idx;
/* Simulate a retried restore (e.g. after a partial earlier failure) --
* same name/admin bits, must find-and-continue, never a duplicate
* create or an -EEXIST-style conflict. */
int rc2 = restore_host_state(&g_tur.c, "retry-host", XCE_WORKER_CLOSED);
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc2,
"retried restore must not error");
worker_t* w2 = tur_find("retry-host");
TEST_ASSERT_NOT_NULL(w2);
TEST_ASSERT_EQUAL_INT_MESSAGE(idx1, w2->idx,
"retry must reuse the same entry, never create a second one");
TEST_ASSERT_TRUE(w2->host.status & XCE_WORKER_CLOSED);
tur_teardown();
printf("test_restore_host_state_retry_is_idempotent: PASS\n");
}
/* ── Hit on a real, already-joined worker: OR bits only (Req 1.10) ──────── */
static void test_restore_host_state_on_real_worker_only_ors_admin_bits(void)
{
tur_setup();
hostattrs_t attrs;
memset(&attrs, 0, sizeof(attrs));
attrs.hosttype = makeString("LINUX");
attrs.hostmodel = makeString("x86");
attrs.cpufactor = 1.0f;
attrs.maxcpus = 4;
attrs.maxmem = 16;
attrs.maxswap = 32;
attrs.maxtmp = 64;
attrs.ndisks = 1;
xce_hostconfig_t hconfig;
memset(&hconfig, 0, sizeof(hconfig));
hconfig.hosttype = (char*)"!";
hconfig.hostmodel = (char*)"!";
hconfig.mxj = 4;
endpoint_t ep;
memset(&ep, 0, sizeof(ep));
int idx = init_worker(&g_tur.c, &ep, "real-host-1", &attrs, &hconfig);
TEST_ASSERT_GREATER_OR_EQUAL(0, idx);
worker_t* w = tur_find("real-host-1");
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_FALSE(w->host.status & XCE_WORKER_UNAVAIL);
TEST_ASSERT_EQUAL_PTR(&ep, w->ep);
int rc = restore_host_state(&g_tur.c, "real-host-1", XCE_WORKER_CLOSED);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_TRUE_MESSAGE(w->host.status & XCE_WORKER_CLOSED,
"admin bit must be OR'd onto the real worker");
TEST_ASSERT_FALSE_MESSAGE(w->host.status & XCE_WORKER_UNAVAIL,
"restore on a real, already-joined worker must not mark it UNAVAIL");
TEST_ASSERT_EQUAL_STRING_MESSAGE("LINUX", w->host.attrs->hosttype,
"restore on a hit must not touch existing attrs");
TEST_ASSERT_EQUAL_PTR_MESSAGE(&ep, w->ep,
"restore on a hit must not touch ep");
tur_teardown();
printf("test_restore_host_state_on_real_worker_only_ors_admin_bits: PASS\n");
}
/* ── Observable (task 3's own line): placeholder -> later real join ────── */
static void
test_restore_host_state_placeholder_then_real_join_preserves_admin_status(void)
{
tur_setup();
/* Host admin-closed while down, then a configuration upgrade restores
* its admin-state frame on a member with no local record of the host's
* join -- must create a placeholder carrying the CLOSED bit. */
int rc = restore_host_state(&g_tur.c, "join-later-host",
XCE_WORKER_CLOSED);
TEST_ASSERT_EQUAL_INT(0, rc);
worker_t* w = tur_find("join-later-host");
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_TRUE(w->host.status & XCE_WORKER_CLOSED);
TEST_ASSERT_TRUE(w->host.status & XCE_WORKER_UNAVAIL);
/* The host's real first join. join.c's own dispatch (join.c:90-103)
* does exactly: key_find() -> found w != NULL -> rejoin_worker() (never
* init_worker()). Reproduced directly here (rather than going through
* handle_worker_join) since that needs a live xce_request_t/endpoint_t/
* persist commit-barrier harness this unit test intentionally has none
* of -- the dispatch decision itself is not under test, join.c's is
* unmodified and already covered elsewhere; what's under test is that
* THIS placeholder is a valid, non-crashing input to rejoin_worker(). */
hostattrs_t new_attrs;
memset(&new_attrs, 0, sizeof(new_attrs));
new_attrs.hosttype = makeString("LINUX");
new_attrs.hostmodel = makeString("x86_64");
new_attrs.cpufactor = 1.0f;
new_attrs.maxcpus = 8;
new_attrs.maxmem = 32;
new_attrs.maxswap = 64;
new_attrs.maxtmp = 128;
new_attrs.ndisks = 2;
xce_hostconfig_t hconfig;
memset(&hconfig, 0, sizeof(hconfig));
hconfig.hosttype = (char*)"!";
hconfig.hostmodel = (char*)"!";
hconfig.mxj = 4;
endpoint_t ep;
memset(&ep, 0, sizeof(ep));
worker_t* found = NULL;
int findrc = 0;
lock(g_tur.c.workermgr, RTTI_LOCK_READ);
findrc = key_find(g_tur.c.workermgr, (void*)"join-later-host",
(void**)&found);
unlock(g_tur.c.workermgr);
TEST_ASSERT_GREATER_THAN_MESSAGE(-1, findrc,
"join.c's dispatch must find the placeholder (hit, not miss)");
TEST_ASSERT_EQUAL_PTR(w, found);
int idx = rejoin_worker(&g_tur.c, found, &ep, &new_attrs, &hconfig);
TEST_ASSERT_GREATER_OR_EQUAL_MESSAGE(0, idx,
"rejoin_worker must not fail on the placeholder");
TEST_ASSERT_TRUE_MESSAGE(w->host.status & XCE_WORKER_CLOSED,
"admin status set via restore must still be set after the real join");
TEST_ASSERT_FALSE_MESSAGE(w->host.status & XCE_WORKER_UNAVAIL,
"the real join must clear UNAVAIL");
TEST_ASSERT_EQUAL_STRING("LINUX", w->host.attrs->hosttype);
dereference(new_attrs.hosttype);
dereference(new_attrs.hostmodel);
tur_teardown();
printf("test_restore_host_state_placeholder_then_real_join_preserves_admin_status: PASS\n");
}
/* ── Defense-in-depth: non-admin bits are masked out ────────────────────── */
static void test_restore_host_state_masks_non_admin_bits(void)
{
tur_setup();
int rc = restore_host_state(&g_tur.c, "masked-host",
XCE_WORKER_CLOSED | XCE_WORKER_BUSY);
TEST_ASSERT_EQUAL_INT(0, rc);
worker_t* w = tur_find("masked-host");
TEST_ASSERT_NOT_NULL(w);
TEST_ASSERT_TRUE_MESSAGE(w->host.status & XCE_WORKER_CLOSED,
"admin bit must be applied");
TEST_ASSERT_FALSE_MESSAGE(w->host.status & XCE_WORKER_BUSY,
"non-admin bit must be masked out even if a caller passes it");
tur_teardown();
printf("test_restore_host_state_masks_non_admin_bits: PASS\n");
}
/* ── Input validation at the boundary ───────────────────────────────────── */
static void test_restore_host_state_rejects_empty_name(void)
{
tur_setup();
TEST_ASSERT_EQUAL_INT(-EINVAL,
restore_host_state(&g_tur.c, NULL, XCE_WORKER_CLOSED));
TEST_ASSERT_EQUAL_INT(-EINVAL,
restore_host_state(&g_tur.c, "", XCE_WORKER_CLOSED));
tur_teardown();
printf("test_restore_host_state_rejects_empty_name: PASS\n");
}
/* ── Dispatch ───────────────────────────────────────────────────────── */
void test_upgrade_snapshot_restore(void)
{
UnitySetTestFile(__FILE__);
printf("\n>> config-upgrade-snapshot host restore tests (task 3)\n\n");
RUN_TEST(test_restore_host_state_creates_placeholder_for_unknown_host);
RUN_TEST(test_restore_host_state_retry_is_idempotent);
RUN_TEST(test_restore_host_state_on_real_worker_only_ors_admin_bits);
RUN_TEST(test_restore_host_state_placeholder_then_real_join_preserves_admin_status);
RUN_TEST(test_restore_host_state_masks_non_admin_bits);
RUN_TEST(test_restore_host_state_rejects_empty_name);
}

134
src/controller/testsuite/test_waldump.c
View file

@ -2,6 +2,8 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>
#include <zlib.h>
@ -152,6 +154,135 @@ static void test_missing_prefix_flagged_as_gap(void)
TEST_ASSERT_EQUAL_UINT64(13, st.first_lsn);
/* prefix [1..12] absent → exactly one gap */
TEST_ASSERT_EQUAL_UINT(1, st.n_gaps);
/* g_dir has no sibling data/snap/ at all → nothing can explain it */
TEST_ASSERT_EQUAL_INT(0, st.leading_gap_explained);
}
/* ------------------------------------------------------------------ */
/* WAL_CONFIG_MARKER must decode as its own named type, not fall into */
/* the generic UNKNOWN bucket (a real, defined type — wal_types.h). */
/* ------------------------------------------------------------------ */
static void test_config_marker_type_decoded(void)
{
fresh_dir();
char path[512];
FILE* fp = fopen(seg_path(1, path, sizeof(path)), "wb");
TEST_ASSERT_NOT_NULL(fp);
uint32_t last = WAL_CRC_NONE;
put_entry(fp, 0, 1, WAL_NOOP, NULL, 0, &last, 1);
uint8_t marker[WAL_CONFIG_MARKER_PAYLOAD_LEN];
uint64_t tag = 0x1122334455667788ULL;
for (int i = 0; i < 8; i++)
marker[i] = (uint8_t)(tag >> (56 - i * 8));
memset(marker + 8, 0xab, 32);
put_entry(fp, 3, 2, WAL_CONFIG_MARKER, marker, sizeof(marker), &last, 0);
fclose(fp);
char* buf = NULL;
size_t bufsz = 0;
FILE* mem = open_memstream(&buf, &bufsz);
TEST_ASSERT_NOT_NULL(mem);
waldump_stats_t st;
int rc = waldump_path(g_dir, mem, &st);
fclose(mem);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_NOT_NULL(strstr(buf, "CONFIG_MARKER"));
TEST_ASSERT_NULL(strstr(buf, "UNKNOWN"));
TEST_ASSERT_NOT_NULL(strstr(buf, "tag=1122334455667788"));
free(buf);
}
/* ------------------------------------------------------------------ */
/* A leading gap covered by a sibling data/snap/ snapshot is normal, */
/* checkpoint-triggered WAL trim — waldump must say so, not alarm. */
/* ------------------------------------------------------------------ */
static void test_leading_gap_explained_by_covering_snapshot(void)
{
fresh_dir();
char wal_dir[300], snap_dir[300];
snprintf(wal_dir, sizeof(wal_dir), "%s/data/wal", g_dir);
snprintf(snap_dir, sizeof(snap_dir), "%s/data/snap", g_dir);
char data_dir[300];
snprintf(data_dir, sizeof(data_dir), "%s/data", g_dir);
TEST_ASSERT_EQUAL_INT(0, mkdir(data_dir, 0755));
TEST_ASSERT_EQUAL_INT(0, mkdir(wal_dir, 0755));
TEST_ASSERT_EQUAL_INT(0, mkdir(snap_dir, 0755));
/* Snapshot at LSN 12 covers the trimmed prefix [1..12] below. */
char snap_file[400];
snprintf(snap_file, sizeof(snap_file), "%s/snap-%016x.capnp", snap_dir,
0xc);
FILE* sf = fopen(snap_file, "wb");
TEST_ASSERT_NOT_NULL(sf);
fclose(sf);
char segp[400];
snprintf(segp, sizeof(segp), "%s/seg-%016lx.wal", wal_dir,
(unsigned long)13);
FILE* fp = fopen(segp, "wb");
TEST_ASSERT_NOT_NULL(fp);
uint32_t last = WAL_CRC_NONE;
put_entry(fp, 0, 13, WAL_NOOP, NULL, 0, &last, 1);
fclose(fp);
char* buf = NULL;
size_t bufsz = 0;
FILE* mem = open_memstream(&buf, &bufsz);
TEST_ASSERT_NOT_NULL(mem);
waldump_stats_t st;
int rc = waldump_path(g_dir, mem, &st);
fclose(mem);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT(1, st.n_gaps);
TEST_ASSERT_NOT_EQUAL(0, st.leading_gap_explained);
TEST_ASSERT_NOT_NULL(strstr(buf, "covered by snapshot"));
TEST_ASSERT_NULL(strstr(buf, "committed prefix missing!"));
free(buf);
}
/* ------------------------------------------------------------------ */
/* A snapshot that does NOT cover the full gap must NOT explain it — */
/* the alarming wording stays, since the gap could still be real loss. */
/* ------------------------------------------------------------------ */
static void test_leading_gap_not_explained_by_insufficient_snapshot(void)
{
fresh_dir();
char wal_dir[300], snap_dir[300], data_dir[300];
snprintf(data_dir, sizeof(data_dir), "%s/data", g_dir);
snprintf(wal_dir, sizeof(wal_dir), "%s/data/wal", g_dir);
snprintf(snap_dir, sizeof(snap_dir), "%s/data/snap", g_dir);
TEST_ASSERT_EQUAL_INT(0, mkdir(data_dir, 0755));
TEST_ASSERT_EQUAL_INT(0, mkdir(wal_dir, 0755));
TEST_ASSERT_EQUAL_INT(0, mkdir(snap_dir, 0755));
/* Snapshot at LSN 5 does NOT cover the trimmed prefix [1..12] below. */
char snap_file[400];
snprintf(snap_file, sizeof(snap_file), "%s/snap-%016x.capnp", snap_dir,
0x5);
FILE* sf = fopen(snap_file, "wb");
TEST_ASSERT_NOT_NULL(sf);
fclose(sf);
char segp[400];
snprintf(segp, sizeof(segp), "%s/seg-%016lx.wal", wal_dir,
(unsigned long)13);
FILE* fp = fopen(segp, "wb");
TEST_ASSERT_NOT_NULL(fp);
uint32_t last = WAL_CRC_NONE;
put_entry(fp, 0, 13, WAL_NOOP, NULL, 0, &last, 1);
fclose(fp);
waldump_stats_t st;
int rc = waldump_path(g_dir, stdout, &st);
TEST_ASSERT_EQUAL_INT(0, rc);
TEST_ASSERT_EQUAL_UINT(1, st.n_gaps);
TEST_ASSERT_EQUAL_INT(0, st.leading_gap_explained);
}
/* ------------------------------------------------------------------ */
@ -288,6 +419,9 @@ void test_waldump(void)
UnitySetTestFile(__FILE__);
RUN_TEST(test_contiguous_log_has_no_gaps);
RUN_TEST(test_missing_prefix_flagged_as_gap);
RUN_TEST(test_config_marker_type_decoded);
RUN_TEST(test_leading_gap_explained_by_covering_snapshot);
RUN_TEST(test_leading_gap_not_explained_by_insufficient_snapshot);
RUN_TEST(test_duplicate_submit_detection);
RUN_TEST(test_interior_gap_flagged);
RUN_TEST(test_crc_break_sets_recovered_tip);

92
src/controller/testsuite/waldump.c
View file

@ -46,6 +46,7 @@ static const char* wal_type_name(uint32_t t)
case WAL_QUEUE_CTRL: return "QUEUE_CTRL";
case WAL_WORKER_CTRL: return "WORKER_CTRL";
case WAL_NOOP: return "NOOP";
case WAL_CONFIG_MARKER: return "CONFIG_MARKER";
case WAL_CHECKPOINT: return "CHECKPOINT";
default: return "UNKNOWN";
}
@ -127,6 +128,63 @@ static int decode_jobstate_info(const uint8_t* payload, uint32_t len,
return 0;
}
/* Decode a WAL_CONFIG_MARKER payload -- fixed 40-byte form (8-byte
* big-endian version_tag + 32-byte raw sha256), never capnp-encoded
* (config_repl.c's config_marker_encode/config_marker_decode). Hand-decoded
* here rather than linking config_repl.c, which would pull controller-
* internal dependencies into xce_test's link graph (meson.build's
* xce_test_src is deliberately limited to read-only dump/compare tools).
* Mirrors config_marker_decode's byte layout exactly. */
static int decode_config_marker_info(const uint8_t* payload, uint32_t len,
uint64_t* out_tag, uint8_t sha256_out[32])
{
if (!payload || len < WAL_CONFIG_MARKER_PAYLOAD_LEN)
return -1;
uint64_t tag = 0;
for (int i = 0; i < 8; i++)
tag = (tag << 8) | payload[i];
*out_tag = tag;
memcpy(sha256_out, payload + 8, 32);
return 0;
}
/* Highest LSN among sibling data/snap/snap-<hex>.capnp files next to
* `wal_dir` (persist.c hardcodes both as siblings under a node's data/,
* e.g. .../data/wal and .../data/snap). A leading WAL gap up to this LSN is
* explained by normal checkpoint-triggered trim (persist.c's
* trim_wal_segments only removes segments below a replication-safe floor,
* never a segment a standby still needs to catch up) rather than genuine
* loss -- without this check, EVERY long-running node's WAL (which will
* have been checkpointed at least once) prints the same alarming
* "(committed prefix missing!)" wording a real, unexplained loss would,
* making the message a standing false alarm. Returns 0 if `wal_dir` isn't
* named "wal", or no sibling snap dir/files exist (unknown -- callers keep
* today's un-downgraded wording, the conservative default). */
static uint64_t highest_snapshot_lsn(const char* wal_dir)
{
char snap_dir[4096];
snprintf(snap_dir, sizeof(snap_dir), "%s", wal_dir);
char* slash = strrchr(snap_dir, '/');
if (!slash || strcmp(slash + 1, "wal") != 0)
return 0;
snprintf(slash + 1, sizeof(snap_dir) - (size_t)(slash + 1 - snap_dir),
"snap");
DIR* d = opendir(snap_dir);
if (!d)
return 0;
uint64_t max_lsn = 0;
struct dirent* e;
while ((e = readdir(d)) != NULL) {
uint64_t lsn;
if (sscanf(e->d_name, "snap-%" SCNx64 ".capnp", &lsn) == 1 &&
lsn > max_lsn)
max_lsn = lsn;
}
closedir(d);
return max_lsn;
}
/* True if `dir` directly contains at least one seg-*.wal file. */
static int dir_has_segments(const char* dir)
{
@ -241,7 +299,7 @@ static int collect_segments(const char* wal_dir, uint64_t* seg_lsns, int max)
static void scan_segment(const char* seg_path, FILE* out,
uint32_t* last_crc, uint64_t* prev_lsn,
uint64_t* prev_submit_jobid, uint32_t* prev_submit_pcrc,
waldump_stats_t* st,
waldump_stats_t* st, uint64_t snap_lsn,
waldump_visit_fn visit, void* visit_ctx,
int* visit_stop)
{
@ -309,10 +367,15 @@ static void scan_segment(const char* seg_path, FILE* out,
/* Gap detection against the running LSN sequence. */
if (*prev_lsn == 0) {
if (hdr.lsn > 1) {
int explained = snap_lsn >= hdr.lsn - 1;
fprintf(out, " *** GAP: missing prefix LSN 1..%" PRIu64
" (%" PRIu64 " entries absent) ***\n",
hdr.lsn - 1, hdr.lsn - 1);
" (%" PRIu64 " entries absent)%s ***\n",
hdr.lsn - 1, hdr.lsn - 1,
explained ? " -- covered by snapshot, expected "
"post-checkpoint trim"
: "");
st->n_gaps++;
st->leading_gap_explained = explained;
}
} else if (hdr.lsn > *prev_lsn + 1) {
fprintf(out, " *** GAP: missing LSN %" PRIu64 "..%" PRIu64 " ***\n",
@ -404,6 +467,20 @@ static void scan_segment(const char* seg_path, FILE* out,
}
/* a non-submit entry breaks the adjacency for dup detection */
*prev_submit_jobid = 0;
} else if (hdr.type == WAL_CONFIG_MARKER) {
uint64_t tag = 0;
uint8_t sha[32];
if (decode_config_marker_info(payload, hdr.payload_len, &tag,
sha) == 0) {
char hex[65];
for (int i = 0; i < 32; i++)
snprintf(hex + i * 2, 3, "%02x", sha[i]);
snprintf(info, sizeof(info), "tag=%016" PRIx64 " sha256=%s",
tag, hex);
} else {
snprintf(info, sizeof(info), "tag=?");
}
*prev_submit_jobid = 0;
} else {
/* a non-submit entry breaks the adjacency for dup detection */
*prev_submit_jobid = 0;
@ -505,6 +582,7 @@ int waldump_path_visit(const char* path, FILE* out, waldump_stats_t* stats,
}
st.n_segments = (uint32_t)nseg;
uint64_t snap_lsn = highest_snapshot_lsn(wal_dir);
fprintf(out, "== WAL: %s (%d segment%s) ==\n", wal_dir, nseg,
nseg == 1 ? "" : "s");
@ -521,7 +599,7 @@ int waldump_path_visit(const char* path, FILE* out, waldump_stats_t* stats,
snprintf(seg_path, sizeof(seg_path), WAL_SEG_FMT, wal_dir,
seg_lsns[i]);
scan_segment(seg_path, out, &last_crc, &prev_lsn,
&prev_submit_jobid, &prev_submit_pcrc, &st,
&prev_submit_jobid, &prev_submit_pcrc, &st, snap_lsn,
visit, ctx, &visit_stop);
}
@ -532,7 +610,11 @@ int waldump_path_visit(const char* path, FILE* out, waldump_stats_t* stats,
" recovered tip: %" PRIu64 "\n",
st.first_lsn, st.last_lsn, st.recovered_tip);
fprintf(out, " gaps: %u%s crc chain: %s\n", st.n_gaps,
(st.n_gaps && st.first_lsn > 1) ? " (committed prefix missing!)" : "",
(st.n_gaps && st.first_lsn > 1)
? (st.leading_gap_explained
? " (explained by checkpoint/snapshot -- not a loss)"
: " (committed prefix missing!)")
: "",
st.crc_broken ? "BROKEN" : "intact");
if (st.n_dup_submits > 0)
fprintf(out, " duplicate submits: %u (byte-identical: %u) — bug-845\n",

7
src/controller/testsuite/waldump.h
View file

@ -29,6 +29,13 @@ typedef struct {
uint32_t n_dup_identical; /* of n_dup_submits, those whose payload is
* byte-identical to the prior entry */
int crc_broken; /* nonzero if a CRC/chain break truncated a segment */
int leading_gap_explained; /* nonzero if the leading (prefix) gap's
* upper LSN is covered by a sibling
* data/snap/snap-<hex>.capnp file — i.e.
* normal checkpoint-triggered WAL trim
* (persist.c trim_wal_segments), not
* genuine loss. 0 if there was no leading
* gap, or no covering snapshot was found. */
} waldump_stats_t;
/* One physically-present, CRC-valid WAL entry, surfaced to a visitor so a

144
src/controller/worker.c
View file

@ -436,6 +436,105 @@ int restore_worker(void* cls, xce_worker_t host)
return idx;
}
/*
* restore_host_state — restore a host admin-state frame (config-upgrade-
* snapshot Requirements 1.10, 1.11, 5.8).
*
* Unlike restore_queue_state() (queues are config-provisioned, so an
* update-only lookup is always safe), a worker_t is join-provisioned: it
* may not exist yet on a member that hasn't locally replayed this host's
* original join record (e.g. one that received the upgrade snapshot via
* the out-of-band install path). So this restore path is create-if-missing:
*
* - Hit (host already registered -- a real joined worker, or an earlier
* placeholder left by a prior restore attempt): OR the restored admin
* bits into the existing status only. Nothing else is touched, so a
* real worker's live attrs/ep/resource values survive untouched.
* - Miss: create a minimal placeholder via makeWorker(), mirroring the
* synthetic lost_and_found sentinel host precedent exactly
* (config_repl.c:1114-1130) -- the same "makeWorker() without a join"
* mechanism, not a new one. The placeholder is marked
* XCE_WORKER_UNAVAIL (unjoined, dead-until-connected) and carries the
* restored admin bits. Its hostattrs_t is non-NULL with valid sentinel
* strings (copy_worker, wobj.c:53, requires non-NULL attrs; downstream
* apply_join_state()/add_hostattrs_resvals() treat hosttype/hostmodel
* as always-valid strings) so the host's later real join -- which
* join.c routes to rejoin_worker(), unmodified, because key_find() now
* finds this placeholder -- doesn't crash on rejoin_worker's direct
* attrs->hosttype dereference (worker.c:376-379).
*
* Idempotent under retry (5.8): find first via our own key_find, create
* only if truly missing. A second call for the same host (e.g. a retried
* restore after an earlier attempt failed partway through the snapshot)
* finds either the real worker or the already-created placeholder and just
* OR's the bits in again -- never a duplicate-create conflict.
*
* admin_status is masked to XCE_WORKER_ADMIN_MASK defensively (mirrors the
* collect side's own discipline, collect_host_frames only ever encoding
* admin-masked bits) so a caller passing an unmasked status word can never
* leak a non-admin bit (e.g. XCE_WORKER_BUSY) onto the restored host.
*/
int restore_host_state(cluster_t* c, const char* name, uint32_t admin_status)
{
worker_t* w = NULL;
int rc = 0;
if (name == NULL || name[0] == '\0')
return -EINVAL;
if (c == NULL || c->workermgr == NULL)
return -EINVAL;
admin_status &= XCE_WORKER_ADMIN_MASK;
/* No reference() pin across the workermgr-unlock/w-lock gap (unlike
* join.c's live-join path): this is called from checkpoint restore
* (ha_handle_snapshot_chunk -> persist_load_snapshot), which DOES run
* on the live vtask pool, not a quiesced phase, so a real first-time
* join for the same host CAN race this call. That race is still safe:
* a concurrent init_worker/makeWorker on a miss here would itself
* find-or-create via the same key_find/makeWorker machinery under
* workermgr's own lock, and either ordering converges to a worker
* with the admin bits OR'd in and UNAVAIL cleared by the real join's
* own apply_join_state() call -- the same "outer key_find, then
* unconditionally call into makeWorker" shape already used elsewhere
* in this codebase (worker_from_capnp/restore_worker,
* create_lost_and_found). Mirrors restore_queue_state's (queue.c:208)
* lock idiom, not a claim that this path is concurrency-free. */
lock(c->workermgr, RTTI_LOCK_READ);
rc = key_find(c->workermgr, (void*)name, (void**)&w);
unlock(c->workermgr);
if (rc < 0 && rc != -ENOENT)
return rc;
if (w != NULL) {
/* Hit: real joined worker or an earlier placeholder -- only OR the
* admin bits in, never touch anything else. */
lock(w, RTTI_LOCK_WRITE);
w->host.status |= admin_status;
unlock(w);
return 0;
}
/* Miss: create a minimal, valid placeholder -- mirrors the
* lost_and_found sentinel host precedent (config_repl.c:1114-1130). */
{
hostattrs_t placeholder_attrs = { .hosttype = "unknown",
.hostmodel = "unknown" };
xce_worker_t placeholder_host = { .name = (char*)name,
.attrs = &placeholder_attrs,
.status = XCE_WORKER_UNAVAIL
| admin_status };
w = makeWorker(c->workermgr, &placeholder_host);
if (w == NULL)
return -errno;
}
return 0;
}
struct worker_config_publish_args {
cluster_t* c;
endpoint_t* ep;
@ -724,12 +823,37 @@ int do_hostcontrol_op(cluster_t* c, endpoint_t* ep, xce_request_t* request)
/* Reconstruct a worker from a WAL_WORKER_CTRL Cap'n Proto payload.
* Payload layout: [uint32_t detail][Cap'n Proto xce_worker_t].
* Returns -EEXIST if the worker is already registered (idempotent replay). */
*
* If the worker is already registered:
* - HOOK_DETAIL_WORKER_JOIN: a duplicate JOIN replay is genuinely a
* no-op -- returns -EEXIST, no mutation (unchanged, pre-bug-1472
* behavior).
* - HOOK_DETAIL_WORKER_UPDATE: the target already existing is the
* NORMAL case (worker_ctrl()'s HOST_OPEN/HOST_CLOSE, worker.c:732-794,
* only ever runs against an already-joined host). bug-1472: replace
* (not OR -- worker_ctrl uses assignment-style clear/set, so a
* HOST_OPEN must be able to CLEAR the closed bits) the existing
* worker's admin-masked status bits with the decoded payload's, then
* return 0 so the caller (persist.c's WAL_WORKER_CTRL apply case)
* falls through to fire the HOOK_DETAIL_WORKER_UPDATE notification
* instead of silently discarding the change.
* - HOOK_DETAIL_WORKER_DEAD: left as -EEXIST (unchanged) -- see
* buglog.json bug-1472 for why DEAD's replication needs its own,
* separately-tracked fix: its mutation isn't confined to
* XCE_WORKER_ADMIN_MASK (XCE_WORKER_UNAVAIL + dead_since + hostattrs
* reset), and the live per-host resource-value array
* (xce_worker_t.resvals, a zarray_t) that mark_worker_unavail()
* zeroes via set_host_slots() never round-trips through the Worker
* capnp schema at all (only resvals_to_sched does) -- a naive
* status-replace here would be an incomplete fix. */
int worker_from_capnp(cluster_t* c, const void* capnp_payload, size_t len)
{
if (!capnp_payload || len <= sizeof(uint32_t))
return -EINVAL;
uint32_t hook_detail;
memcpy(&hook_detail, capnp_payload, sizeof(hook_detail));
/* Skip the 4-byte detail discriminator written by persist_worker_hook_fun */
const uint8_t* data = (const uint8_t*)capnp_payload + sizeof(uint32_t);
size_t data_len = len - sizeof(uint32_t);
@ -745,13 +869,27 @@ int worker_from_capnp(cluster_t* c, const void* capnp_payload, size_t len)
decode_Worker(&host, &cw);
capn_free(&c_ctx);
/* Idempotency guard: skip if already present */
/* Idempotency guard: an existing worker means either a duplicate JOIN
* replay (no-op) or an UPDATE targeting an already-joined host (the
* normal case -- must be applied, see bug-1472). */
worker_t* existing = NULL;
lock(c->workermgr, RTTI_LOCK_READ);
int found = key_find(c->workermgr, (void*)host.name, (void**)&existing);
unlock(c->workermgr);
if (found > 0 && existing != NULL)
if (found > 0 && existing != NULL) {
if (hook_detail == HOOK_DETAIL_WORKER_UPDATE) {
lock(existing, RTTI_LOCK_WRITE);
existing->host.status
= (existing->host.status & ~XCE_WORKER_ADMIN_MASK)
| (host.status & XCE_WORKER_ADMIN_MASK);
unlock(existing);
free_Worker(&host);
return 0;
}
return -EEXIST;
}
return restore_worker(c, host);
}

255
src/docs/deployment.md Normal file
View file

@ -0,0 +1,255 @@
# XCE Deployment Guide
Covers building and running all four daemons (controller, worker, scheduler, apiserver) and using `xceadmin` to author and push cluster configuration. Local dev/test deployment is fully automated; production process supervision (systemd/Docker/etc.) is **not** shipped in this repo — see [Production Gaps](#production-gaps) before treating anything here as a production runbook.
## Component Overview
```
xceadmin (Python CLI) --config push (TLS)--> controller (C, HA cluster, 1..N nodes)
^ ^
plaintext join | | plaintext join
| |
worker (C) scheduler (C)
^
CGO-linked libxce client (TCP, HA-aware)
|
apiserver (Go, REST) <-- xcecli (Go CLI, REST)
```
- **controller** (`xce_controller`) — the only stateful daemon. Runs as 1 or 3+ nodes for HA (Raft-style election + replication). All other components are its clients.
- **worker** (`xcecd`) — one per compute host, runs jobs, joins a controller.
- **scheduler** (`xce_scheduler`) — job-to-node placement engine, joins a controller.
- **apiserver** (`xce_apiserver`) — Go REST API, joins the controller cluster as a client role (`HostRole_apiserver`) via a CGO-linked `libxce` client, not a plain HTTP backend call.
- **xcecli** — Go CLI, talks to `apiserver` over REST, never touches the controller directly.
- **xceadmin** — Python CLI, the only tool that compiles and pushes *cluster configuration* (`cluster.bin`) to the controller. Separate concern from the four daemons above.
Only controller↔controller HA replication traffic is TLS. Worker→controller, scheduler→controller, and apiserver→controller traffic is **plaintext**.
## Building
```bash
# Dev loop, per component (debug + ASAN, runs tests)
/xce-build controller
/xce-build worker
/xce-build scheduler
/xce-build apiserver
/xce-build xceadmin
/xce-build # all components
# Profiles: /xce-build [component] [profile]
# asan (default) -> xce.build/ debug+ASAN, runs tests
# debug -> build.dbg/ -O0 -g3, no sanitizer, gdb-friendly
# release -> build.release/ -O3, no sanitizer, build-only
# Production (Nix)
nix build .#xcecore # controller + worker + scheduler + apiserver + xceadmin + AppImage, one derivation
nix build .#xcecore_debug # same, debug + ASAN (this flake's default package)
nix build .#xcecli # xcecli only (separate Go-only derivation)
```
`nix build .#xcecore` is all-or-nothing — there is no per-daemon Nix target. It produces one `result/bin/` containing `xce_controller`, `xcecd` (+ `xcecd_worker`, `xcecd_virtual`), `xce_scheduler`, `xce_apiserver`, the `xceadmin` venv wrapper, and `xce` (the AppImage dispatcher script, see below). `nix build .#xcecli` is the one genuinely separate target.
**AppImage** (`xce-x86_64.AppImage`, from `nix build .#xcecore`) wraps `src/tools/appimage/entry.sh`, a thin per-daemon dispatcher — it runs exactly **one** daemon per invocation, never the whole stack:
```bash
./xce-x86_64.AppImage controller --myname ctrl-1 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990,ctrl-3=h3:9990" ...
./xce-x86_64.AppImage worker -c /etc/xce/worker.yaml
./xce-x86_64.AppImage scheduler --peers node1=h1:9990
./xce-x86_64.AppImage apiserver -d /opt/xce/apiserver
./xce-x86_64.AppImage shell # interactive bash with all binaries + haproxy/glow/less on PATH
```
Run it once per daemon (likely on different hosts) to stand up a cluster.
## xceadmin — Cluster Configuration
`xceadmin` is the only supported way to turn operator-authored YAML into the `cluster.bin` binary the controller loads, and to push it to a running cluster. It speaks the same raw TLS + Cap'n Proto wire protocol controllers use with each other (no client certificate — `verify_mode=ssl.CERT_NONE`, matching the controller's own outbound posture).
### Install
```bash
# Dev, editable, fast iteration (must run inside `nix develop ./src`)
cd src/xceadmin && pip install -e .
# Production / CI (what /xce-build and `nix build .#xcecore` actually use)
/xce-build xceadmin
```
`/xce-build xceadmin` always does a clean, non-editable `pip install` into an out-of-tree venv (`--system-site-packages`, inheriting nix-provided `pycapnp`/`pyyaml`/`jsonschema`), runs the full pytest suite, syncs the bundled schemas fresh from `src/schema/`, and produces a standalone `xceadmin` wrapper script — this is what ships in the AppImage and `nix build` output.
**This venv only works from inside `nix develop ./src` (or the AppImage's bundled env) — it is not a portable, standalone venv.** `--system-site-packages` means `pip install` sees `click`/`pycapnp`/`pyyaml`/`jsonschema` already satisfied by nix's site-packages and skips vendoring its own copies — deliberately, since `pycapnp` is ABI-sensitive to the exact `capnproto` C library version, and letting pip resolve/build its own would risk a version mismatch that crashes at runtime. If you `source .venv/bin/activate` (or run the `xceadmin` binary directly) outside `nix develop`, none of those packages are on `sys.path` and you'll hit `ModuleNotFoundError: No module named 'click'` (or similar) immediately. Either stay inside `nix develop ./src` when invoking the built `xceadmin`, or use the AppImage, which bundles a working environment on `PATH` already.
### Usage
```bash
# Compile YAML config into a binary, print its SHA-256 to stderr
xceadmin config build /path/to/configs -o cluster.bin
xceadmin config build /path/to/configs -o - # to stdout instead of a file
# Push a compiled config to a running controller (any node — primary or standby; redirects to primary automatically, up to 3 hops)
xceadmin config push cluster.bin --controller 127.0.0.1:9991 # host:port; defaults to localhost:9991
# Check whether the controller would accept it, without committing
xceadmin config push --dry-run cluster.bin --controller 127.0.0.1:9991
```
`config build` reads a fixed set of source files from the given directory — `xce.yaml`, `hosts.yaml`, `queues.yaml`, `resources.yaml`, `users.yaml`, `limits.yaml`, `scaler.yaml` (a missing file → empty section, not an error) — validates each against a bundled JSON schema, runs hard-fail business checks (reserved names, duplicates) and warn-only checks (undefined cross-references), then maps everything into a `ClusterConfig` Cap'n Proto message. Building twice from identical input is byte-for-byte deterministic (no wall-clock/hostname ever read). Fields that are **never** settable from YAML: `port`, hash-table-size tuning, `xstrings`, `candidates` (peer membership), `configGeneration` (controller-assigned).
`config push` exits non-zero unless the controller's reply reports `ackedCount > 0` and no error string; `--dry-run` exits non-zero on any rejection reason the controller returns.
### Known limitations (as of this writing)
- **`config push` against a real controller fails with a client-side decode crash**, live-verified against a freshly deployed single-node controller: the controller's `OPCode_upgradeConfig` dispatch is fine — `CONF_ENABLE_UPGRADE_CONFIG` defaults on, the earlier dispatch-gate polarity bug (bug-1468) was already fixed 2026-07-25, and the controller log shows the push genuinely committed, activated, and hash-verified server-side. The bug is in `xceadmin` itself: `xce_reply()` (`src/lib/request.c`) wraps every reply — including the upgradeConfig reply — in an outer `Request{opcode: OPCode_reply, body.replymsg: RequestReply{...}}` envelope, but `discovery.py`'s `_redirect_target()` decodes the raw wire bytes directly as a bare `RequestReply`, skipping the outer envelope. The resulting pointer-offset misalignment throws `capnp.lib.capnp.KjException: ... non-list pointer where text was expected` from `push_config()`. This affects every opcode reply routed through `follow_redirects`, not just upgradeConfig. Fix belongs in `xceadmin`, not the controller.
- **`config push --dry-run`'s "OK" result is not meaningful evidence of anything**: `probe_config()`'s docstring itself notes `OPCode_probeConfigTag` has no dispatch case in `cluster.c`, so the "accepted" response is a client-side inference, not a real wire signal from the controller. Live-verified: dry-run returns exit 0 with no corresponding entry in the controller log at all.
- No `config diff`, no rollback, no version history command. `xceadmin` today has exactly two subcommands: `config build` and `config push [--dry-run]`.
- No client authentication — the trust model is "same posture as controller-to-controller TLS," i.e. encryption without identity verification.
## Deploying the Controller
The controller is the only daemon with real config-file/CLI complexity for cluster topology.
```bash
xce_controller \
--myname ctrl-1 \
--peers "ctrl-1=host1:9990,ctrl-2=host2:9990,ctrl-3=host3:9990" \
--workdir /var/lib/xce/controller \
--certdir /var/lib/xce/certs \
--logfile /var/log/xce/ctrl-1.log \
--vcpus 4
```
- Omit `--peers` for a single-node, non-HA controller.
- `--workdir` holds per-node `conf/` (cluster config) and `data/` (WAL + checkpoint) subdirectories.
- `--certdir` must contain `xce.crt`/`xce.key` — this is the **only** TLS material in the whole stack, used exclusively for controller↔controller HA traffic. Generate a self-signed pair for local/dev use:
```bash
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
-keyout /var/lib/xce/certs/xce.key -out /var/lib/xce/certs/xce.crt -subj "/CN=xce"
```
TLS verification is `SSL_VERIFY_NONE` everywhere in this codebase — no hostname/identity check is enforced regardless of CA, so a CA-signed cert buys you encryption, not authentication.
- Exactly one node (conventionally the first started) needs its `conf/` seeded with an initial `cluster.bin` (via `xceadmin config build`) before first boot; other nodes receive config over the wire via config-push once joined.
- **WAL and snapshot retention in `data/` is fully automatic — no manual purging needed.** Every checkpoint trims WAL segments below a replication-safe floor (a segment is never deleted while any standby still needs it to catch up) and rotates old snapshots down to `snap_keep_n` most recent (default 2 — current + one previous, kept as a last-known-good fallback if a checkpoint is later found corrupt). `snap_keep_n` is tunable via the cluster config's `PersistConfig.snapKeepN` field, not a CLI flag.
### Local dev cluster (fully automated)
```bash
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
```
This is the `xce-deploy-ha` skill wrapping `src/tools/deploy-ha.sh`, which does everything above automatically: generates certs, builds an initial config from `src/configs/*.yaml` via `xceadmin`, and writes `scripts/start.sh ctrl-N [extra flags]` / `scripts/cleanup.sh` under the deploy directory. Start nodes with a short stagger for clean election ordering:
```bash
scripts/start.sh ctrl-1 -vvv &
sleep 0.5
scripts/start.sh ctrl-2 -vvv &
sleep 0.5
scripts/start.sh ctrl-3 -vvv &
```
(`-vvv` is a **positional** arg to `start.sh`, not `VERBOSE=-vvv` — the skill's own docs have this wrong.)
`scripts/cleanup.sh` wipes logs and every node's `data/`, plus `conf/` for nodes 2..N — ctrl-1's config and all certs survive a cleanup, giving a clean restart without regenerating TLS material or re-authoring config.
## Deploying the Worker
```bash
xcecd -c /opt/xce/conf/worker.yaml
# or, without a config file:
xcecd -m controller-host -p 9990 --peers "ctrl-1=h1:9990,ctrl-2=h2:9990" -l /var/log/xce
```
- **Must run as root** — `check_user()` exits immediately otherwise.
- Default config path is `/opt/xce/conf/worker.yaml` if `-c` is omitted (and it must exist, or the daemon exits).
- `-c`, if given, **wins over** `-m`/`-p` (those flags are ignored with a warning).
- Minimal `worker.yaml`:
```yaml
workerconf:
management_list: ["ctrl-1-host"]
controller_port: 9990
log_level: "info"
log_dir: "/var/log/xce"
```
- `--peers "[name=]host:port,..."` enables HA-aware controller discovery: the worker tries each candidate in turn (every 5s by default) until one accepts the join.
- No TLS on this connection — worker↔controller traffic is plaintext.
## Deploying the Scheduler
```bash
xce_scheduler --peers "ctrl-1=controller-host:9990" -l /var/log/xce/scheduler.log -v
```
- No config file at all — pure CLI (`argp`).
- **Gotcha**: the default controller target is hard-coded to `localhost:9990`. There is no `-m`/`-p` equivalent — the *only* way to point the scheduler at any other host or port, even a single non-HA controller, is `--peers`. Forgetting `--peers` silently targets `localhost:9990` regardless of where your controller actually is.
- `-v` is repeatable for more verbose logging; default log file is `/tmp/xce_scheduler.log`.
- No TLS — plaintext to the controller, same as the worker.
- Scheduling policies (fcfs, limit, parallel/sequential-job allocators) are compiled into the binary, not dynamically loaded plugins.
## Deploying apiserver
```bash
apiserver -d /opt/xce/apiserver
```
`-d <top_dir>` is **mandatory** despite the `--help` text implying a default — omitting it exits immediately. `<top_dir>/config/` must already exist and contain:
| File | Purpose |
|---|---|
| `app.yaml` | HTTP(S) port, JWT secret/expiry, rate limits, CORS, IP whitelist, and the `backend:` block (`address`, `port`, `from`, `key`, optional `peers` for HA) pointing at the controller. |
| `auth.yaml` | Auth provider selection (`config_file` \| `pam` \| `ldap` \| `oauth2`); for `config_file`, embeds usernames/passwords/roles directly. |
| `rbac.yaml` | Permission catalog + role→permission mappings (domains: `self`, `self_ug`, etc.). User→role binding comes from `auth.yaml`, not this file. |
| `logger.yaml` | zap logger level/encoding/rotation. |
Both `app.yaml` and `logger.yaml` are JSON-Schema-validated at startup (fail-fast on malformed config).
### Build gotcha — use the Meson/Nix build, not `make build-server`
`make build-server` (`src/frontend/Makefile`) runs a bare `go build` with **no `CGO_CFLAGS`/`CGO_LDFLAGS` set**. The real controller-connection code (`backend/clib.go`) is CGO and needs those flags to find `libxce`'s headers. Depending on your environment this either fails to compile, or — if `CGO_ENABLED=0` is ambient — silently links `backend/stub.go` instead, producing a binary where every backend call returns `errNoCGO: backend requires CGO`. There is no startup warning; the HTTP server comes up fine and only fails at request time.
**Use `/xce-build apiserver` (or `nix build .#xcecore`)** for anything that needs to actually talk to a controller — it computes the correct `CGO_CFLAGS`/`CGO_LDFLAGS` from pkg-config and produces `xce_apiserver` under `src/xce.build/frontend/apiserver/` (or the equivalent path for other profiles). Reserve `make build-server`/`make test` for the pure-Go unit-test loop (`CGO_ENABLED=0` is *intentional* there).
### Relationship to the controller
apiserver is a **separate OS process**, network-connected to the controller — not embedded, not in-process. It joins the controller's cluster protocol as `HostRole_apiserver` via a CGO-linked `libxce` client (the same join mechanism workers/schedulers use), configured entirely by `app.yaml`'s `backend:` block. If `backend.address` is empty, apiserver logs "running in stub mode" and serves HTTP, but every job/host/queue/cluster handler will fail. No TLS on this connection.
### TLS
apiserver can self-terminate HTTPS (`app.yaml`'s `security.https.enabled` + `cert_file`/`key_file`, default a checked-in self-signed dev cert). A reverse-proxy pattern (sample `config/nginx.https.conf`, HAProxy mentioned in steering docs) is the intended production direction but is **not wired into any startup automation** — you'd have to set it up yourself.
## Putting It Together: A Full Local Cluster
```bash
# 1. Build everything
/xce-build all
# 2. Bring up a 3-node HA controller cluster
/xce-deploy-ha localhost:9991 localhost:9992 localhost:9993
# (start ctrl-1, then ctrl-2/ctrl-3 with a 0.5s stagger, per the deploy script)
# 3. Start a worker pointed at the cluster
xcecd -m localhost -p 9991 --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -l /tmp/xce-worker
# 4. Start a scheduler pointed at the cluster
xce_scheduler --peers "ctrl-1=localhost:9991,ctrl-2=localhost:9992,ctrl-3=localhost:9993" -v
# 5. Start apiserver (needs its own config/ dir prepared first — copy from src/frontend/apiserver/config/,
# at minimum set backend.address/port to match the controller and use auth.yaml.example, not the checked-in auth.yaml)
apiserver -d /path/to/apiserver-topdir
# 6. Use xcecli against apiserver, or xceadmin against the controller directly
xcecli job submit myjob.json
xceadmin config push cluster.bin --controller localhost:9991
```
## Production Gaps
Be aware of these before treating anything above as a production runbook — they are honest gaps in this repo today, not documented-but-hidden features:
1. **No process supervision is shipped.** No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons. A systemd-unit *sketch* exists in `src/docs/config-replication-design.md` but has a wrong binary name (`xcecontroller` vs the real `xce_controller`) and specifies `Type=notify`, which the controller does not support (it never calls `sd_notify`) — using it as written will hang at startup and time out. If you adapt it, switch to `Type=simple` and fix the binary name.
2. **`src/frontend/apiserver/config/auth.yaml` ships plaintext dev credentials checked into git**, including an `admin`/`admin` account and a named-user `cluster_admin` account. Treat all of them as exposed by default — don't ship the checked-in file as-is; copy `auth.yaml.example` and set real credentials/an external auth provider instead.
3. **Only controller↔controller traffic is encrypted.** Worker→controller, scheduler→controller, and apiserver→controller are all plaintext, with no path to enable TLS on them today.
4. **apiserver's own TLS uses a checked-in self-signed cert with no CA trust story**; `xcecli` hardcodes `InsecureSkipVerify: true` against it. A reverse-proxy-terminated-TLS model is sketched (nginx sample config) but not automated.
5. **`xceadmin config push` doesn't work against a real cluster yet** (see xceadmin's Known Limitations above) — the controller side is fine; `xceadmin`'s `discovery.py` fails to unwrap the `Request` envelope around every reply, crashing on decode.
6. **`nix build .#xcecore` is all-or-nothing** — there's no way to build just one daemon via Nix; use `/xce-build <component>` for per-component builds.
## Quick Reference
| Daemon | Binary | Config | Controller-pointing flag | TLS |
|---|---|---|---|---|
| Controller | `xce_controller` | none (CLI only) | `--peers` (omit for single-node) | Yes, controller↔controller only |
| Worker | `xcecd` | `worker.yaml` (or CLI) | `-m`/`-p` (config file wins) or `--peers` | No |
| Scheduler | `xce_scheduler` | none (CLI only) | `--peers` (**required** even for a single non-default controller) | No |
| apiserver | `xce_apiserver` | `config/{app,auth,rbac,logger}.yaml` | `app.yaml`'s `backend:` block | Optional, self-terminated |
| xceadmin | `xceadmin` (Python) | none — takes YAML source dir | `--controller host:port` | Yes, no client cert |

4
src/frontend/apiserver/config/auth.yaml
View file

@ -11,8 +11,8 @@ config_file:
- username: admin
password: admin
roles: [cluster_admin]
- username: rshen
password: Letmein
- username: op7x2k9
password: CHANGE_ME_dev_only
roles: [cluster_admin]
- username: user1
password: user1

8
src/include/client.h
View file

@ -95,6 +95,14 @@ extern int xcecli_jobctrl(xcecli_ctx_t, xce_jobctrl_t*);
extern int xcecli_notify_jobchkpt(xcecli_ctx_t, uint64_t, char*);
extern int xcecli_queuectrl(xcecli_ctx_t, xce_queuectrl_t*);
extern int xcecli_hostctrl(xcecli_ctx_t, xce_hostctrl_t*);
/* xcecli_upgradeconfig: submit a configuration upgrade (sha256 + serialised
* ClusterConfig blob) to the primary via OP_upgradeConfig, transparently
* surviving a primary change mid-call via XCECLI_FAILOVER_CALL (same
* write-op failover contract as xcecli_hostctrl/xcecli_queuectrl). Returns
* the server's reply status (0 on success). */
extern int xcecli_upgradeconfig(xcecli_ctx_t, const uint8_t *sha256,
size_t sha256_len, const uint8_t *bytes,
size_t bytes_len);
extern int xcecli_runtask(xcecli_ctx_t, xce_taskrun_t*);
extern int xcecli_notify_taskstart(xcecli_ctx_t, xce_task_pidinfo_t*);
extern int xcecli_notify_taskfail(xcecli_ctx_t, xce_task_failinfo_t*);

11
src/include/myconf.h
View file

@ -10,6 +10,7 @@
#include <tlog.h>
#include "clusterconfig.h"
#include "mylock.h"
#include "xce/cluster.h"
/* Forward declaration — avoids circular include with cluster_def.h */
@ -50,6 +51,16 @@ typedef struct {
uint32_t resmgr_inst_delta; /* --resmgr-inst-delta CLI value; 0 = use hardcoded default */
uint32_t xstrings_nhash; /* --xstrings-nhash CLI value; 0 = use hardcoded default */
uint32_t xstrings_delta; /* --xstrings-delta CLI value; 0 = use hardcoded default */
/* bug-1476: serializes load_primary_config()'s free-decode-publish
* sequence against concurrent invocations for the SAME myconf_t (e.g.
* two racing handle_repl_config_push deliveries dispatched onto separate
* VTASK_LIVE_BLOCK workers for the same standby -- a live ASan
* heap-use-after-free was caught this way). Must be mylock_init()'d
* before first use (main.c) and mylock_fini()'d in myconf_fini(); test
* fixtures that drive the real load_primary_config()/
* config_activate_committed() must init it too (see
* t43_attach_myconf/t43_free_myconf in test_config_repl.c). */
mylock_t lock;
} myconf_t;
extern cluster_config_t* get_default_conf(void*);

18
src/include/persist.h
View file

@ -660,12 +660,18 @@ uint64_t persist_entry_term_locked(const persist_t* p, uint64_t lsn);
* checkpoint against this member again, so only the checkpoint LSN's own
* term needs reconciling — not the full compacted history below it.
*
* `checkpoint_term == 0` means a legacy sender (pre-task-3.2 peer) never
* stamped the wire field; treat that as WAL_LEGACY_TERM per the existing
* legacy-0 convention (record_entry_term stores the raw value verbatim; the
* legacy-0 -> WAL_LEGACY_TERM mapping is applied uniformly at read-back by
* persist_entry_term/entry_term_locked, so this call is consistent with that
* convention without needing to special-case it here).
* `checkpoint_term == 0` is unambiguous and means the checkpoint LSN is
* genuinely/permanently absent from the term index, never "legacy term 0,
* present" (bug-1475 Finding 2: the sole sender, ha_snapshot_do_send, always
* computes this value via persist_entry_term, whose own contract already
* returns the distinct WAL_LEGACY_TERM for any present-but-legacy-0 entry —
* so a wire value of exactly 0 can only mean absent). This call substitutes
* TERM_INDEX_UNSET for a wire 0 before recording, instead of the raw value,
* so entry_term_locked reads the checkpoint LSN back as absent on the
* receiver too, matching the sender — recording the raw 0 here (the old
* behavior) let it read back as WAL_LEGACY_TERM instead, permanently
* diverging from the sender's own absent(0) and wedging the next
* AppendEntries whose prevLogIndex lands on this LSN.
*
* Takes p->lock internally (callers must NOT already hold it). Safe on a
* NULL p or checkpoint_lsn == 0 (no-op). */

10
src/include/repl.h
View file

@ -137,6 +137,16 @@ void repl_start_catchup(replicator_t *r, repl_ep_t ep, uint64_t join_applied_lsn
/* Mark standby catch-up complete (called on OP_replCatchUpDone receipt) */
void repl_complete_catchup(replicator_t *r, repl_ep_t ep);
/* bug-1479: look up whether the given standby is currently in catch-up and,
* if so, its corrected resume point (join_applied_lsn). Returns 1 and sets
* *join_applied_lsn_out when in_catch_up is set; returns 0 (leaves
* *join_applied_lsn_out untouched) otherwise. Lets a caller (the heartbeat-
* driven replication decision) use the corrected position instead of a
* standby's own self-reported LSN, which is unreliable for a standby stuck
* rejecting every AppendEntries due to a log conflict. */
int repl_get_catchup_join_lsn(replicator_t *r, repl_ep_t ep,
uint64_t *join_applied_lsn_out);
/* Returns 1 if standby needs a full snapshot transfer (its LSN predates oldest WAL).
* oldest_wal_lsn == 0 means no WAL exists — no snapshot needed for LSN 0 standbys. */
int repl_needs_snapshot(replicator_t *r, uint64_t standby_lsn, uint64_t oldest_wal_lsn);

13
src/include/request.h
View file

@ -185,6 +185,19 @@ extern int xce_upgrade_config_req_extract_raw(const uint8_t *pkt_data,
size_t *out_len,
uint8_t **out_sha256);
/*
* xce_upgradeconfig_req_raw: send OP_upgradeConfig carrying raw byte arrays
* (sha256 + the serialised ClusterConfig blob). Bypasses the generated
* encode_UpgradeConfigRequest path (a no-op for the sha256/bytes Data fields
* — same class of limitation as xce_replconfigpush_req_raw) by constructing
* the capnp message directly via capn_new_list8 + capn_setv8 + write_Request.
* Ordinary CLIENT request: seq/from are populated the same way xce_request()
* populates them for every other client op.
*/
extern int xce_upgradeconfig_req_raw(endpoint_t *ep,
const uint8_t *sha256, size_t sha256_len,
const uint8_t *bytes, size_t bytes_len);
/* ── Client-Side Transparent Retry tunables (Req 6.1, 6.2) ──────────────────
*
* xce_retry_config_t: configurable retry parameters with safe production

21
src/include/typemap.h
View file

@ -443,6 +443,25 @@ typedef struct {
uint32_t prevCrc;
} capnp_repl_envelope_t;
/* ── Upgrade snapshot admin-state frames (config-upgrade-snapshot task 1,
* Req 1.9/5.7) ───────────────────────────────────────────────────────────
* Frame-only wire types for the checkpoint's host/queue admin-state
* sections -- carry only the administratively-set (name, status) pair,
* never the full xce_worker_t/xce_queue_t (design.md Design Invariant 5: a
* snapshot frame must never inherit a live struct's dynamic fields just
* because reusing it is convenient). Neither has a corresponding "live" C
* struct, so each gets its own frame-only counterpart here, same as
* capnp_snap_hdr_t/capnp_persist_cfg_t/capnp_repl_envelope_t above. */
typedef struct {
char* name;
uint32_t status;
} capnp_host_admin_state_t;
typedef struct {
char* name;
uint32_t status;
} capnp_queue_admin_state_t;
/* ── Config replication Cap'n Proto mapped types (Phase 1) ──────────────
* C-side counterparts for the capnp encode/decode shims for config sync.
* Struct tags match the forward declarations near the top of this file. */
@ -472,6 +491,8 @@ struct repl_config_push_req_s {
struct repl_config_push_reply_s {
int accepted; /* 1 if standby accepted the push */
uint64_t version_tag; /* standby's current version tag after decision */
uint64_t term; /* replying member's current term (raft-adapter-
* conformance Req 7.4 term-discovery channel) */
};
struct upgrade_config_req_s {

11
src/include/xceapi.h
View file

@ -28,6 +28,17 @@ extern int xce_update_resvals(xce_ctx_t*, int, xce_resval_t**);
extern int xce_jobctrl(xce_ctx_t*, xce_jobctrl_t*);
extern int xce_queuectrl(xce_ctx_t*, xce_queuectrl_t*);
extern int xce_hostctrl(xce_ctx_t*, xce_hostctrl_t*);
/* xce_upgradeconfig: send OP_upgradeConfig (sha256 + serialised ClusterConfig
* blob) via xce_upgradeconfig_req_raw and wait for the reply. Returns the
* server's reply status (0 on success; a negative errno/E_XCE_* code on
* failure — the server logs the human-readable reason, see
* do_upgrade_config_with_bytes/send_upgrade_error, config_repl.c). Mirrors
* xce_hostctrl/xce_queuectrl's shape; the richer UpgradeConfigReply fields
* (version_tag/acked_count/error text) are not surfaced to the caller — no
* consumer of this API currently needs them beyond success/failure. */
extern int xce_upgradeconfig(xce_ctx_t*, const uint8_t *sha256,
size_t sha256_len, const uint8_t *bytes,
size_t bytes_len);
extern int xce_runtask(xce_ctx_t*, xce_taskrun_t*);
extern int xce_update_jusage(xce_ctx_t*, uint64_t, int, int*, int, int*, int,
xce_rusage_t**);

19
src/lib/client.c
View file

@ -778,6 +778,25 @@ int xcecli_hostctrl(xcecli_ctx_t c, xce_hostctrl_t* ctrlinfo)
return rc;
}
int xcecli_upgradeconfig(xcecli_ctx_t c, const uint8_t *sha256,
size_t sha256_len, const uint8_t *bytes,
size_t bytes_len)
{
api_ctx_t* ctx = (api_ctx_t*)c;
int rc;
if ((ctx == NULL) || (sha256 == NULL) || (sha256_len == 0)
|| (bytes == NULL) || (bytes_len == 0)) {
return -EINVAL;
}
rc = XCECLI_FAILOVER_CALL(ctx, /*is_write=*/true,
xce_upgradeconfig(&(ctx->ctx), sha256,
sha256_len, bytes,
bytes_len));
return rc;
}
int xcecli_runtask(xcecli_ctx_t c, xce_taskrun_t* info)
{
api_ctx_t* ctx = (api_ctx_t*)c;

88
src/lib/request.c
View file

@ -499,6 +499,94 @@ int xce_replconfigpush_req_raw(endpoint_t *ep, xce_peerinfo_t *from,
return (enq_rc < 0) ? enq_rc : 0;
}
/*
* xce_upgradeconfig_req_raw: send OP_upgradeConfig with raw byte arrays.
*
* Bypasses encode_UpgradeConfigRequest (a literal no-op for the sha256/bytes
* Data fields — see the doc comment above upgrade_config_req_s in typemap.h;
* confirmed by inspection of the generated xce.build/lib/controller.capnp.c).
* Instead, constructs the struct Request / struct UpgradeConfigRequest
* directly using the capnp wire-format API (capn_new_list8 + capn_setv8),
* mirroring xce_replconfigpush_req_raw's exact bypass pattern.
*
* Unlike that controller-to-controller helper, this is an ordinary CLIENT
* request: seq and from are populated the same way the generic xce_request()
* populates them for every other client op (endpoint_getseq(ep), ep->peer),
* not passed in by the caller.
*/
int xce_upgradeconfig_req_raw(endpoint_t *ep,
const uint8_t *sha256, size_t sha256_len,
const uint8_t *bytes, size_t bytes_len)
{
struct capn c;
struct capn_segment *cs;
struct Request rq;
struct UpgradeConfigRequest up_req;
UpgradeConfigRequest_ptr up_ptr;
Request_ptr rp;
io_packet_t *pkt;
int rc;
if (ep == NULL)
return -EINVAL;
if (sha256 == NULL || sha256_len == 0 || bytes == NULL || bytes_len == 0)
return -EINVAL;
capn_init_malloc(&c);
cs = capn_root(&c).seg;
memset(&rq, 0, sizeof(rq));
memset(&up_req, 0, sizeof(up_req));
/* sha256: write as a List(UInt8) — capn_list8 and capn_data share {capn_ptr p} */
{
capn_list8 sha_list = capn_new_list8(cs, (int)sha256_len);
capn_setv8(sha_list, 0, sha256, (int)sha256_len);
up_req.sha256.p = sha_list.p;
}
/* config bytes */
{
capn_list8 bytes_list = capn_new_list8(cs, (int)bytes_len);
capn_setv8(bytes_list, 0, bytes, (int)bytes_len);
up_req.bytes.p = bytes_list.p;
}
up_ptr = new_UpgradeConfigRequest(cs);
write_UpgradeConfigRequest(&up_req, up_ptr);
rq.opcode = OPCode_upgradeConfig;
rq.seq = endpoint_getseq(ep);
rq.body_which = Request_body_upgradeconfigrequestmsg;
rq.body.upgradeconfigrequestmsg = up_ptr;
encode_PeerInfo_ptr(cs, &rq.from, ep->peer);
rp = new_Request(cs);
write_Request(&rq, rp);
capn_setp(capn_root(&c), 0, rp.p);
pkt = (io_packet_t *)calloc(1, sizeof(*pkt));
if (!pkt) {
capn_free(&c);
return -ENOMEM;
}
pkt->sz = capn_size(&c);
rc = alloc_packet(pkt);
if (rc < 0) {
free(pkt);
capn_free(&c);
return -errno;
}
capn_write_mem(&c, (uint8_t *)pkt->data, pkt->sz, 0);
capn_free(&c);
int enq_rc = endpoint_enq(ep, &(pkt->link));
return (enq_rc < 0) ? enq_rc : 0;
}
/* wal_entry_to_wire: fill a wire WalCatchupEntry from a C entry's header
* fields and raw payload bytes. Sub-entry nesting is one level deep —
* elements written here never carry their own entries list. */

30
src/lib/xceapi.c
View file

@ -647,6 +647,36 @@ int xce_hostctrl(xce_ctx_t* ctx, xce_hostctrl_t* ctrl)
return status;
}
int xce_upgradeconfig(xce_ctx_t* ctx, const uint8_t *sha256,
size_t sha256_len, const uint8_t *bytes,
size_t bytes_len)
{
int rc;
int status;
if (ctx == NULL) {
return -EINVAL;
}
if (ctx->state != XCE_CTX_RUN) {
return -EINVAL;
}
rc = xce_upgradeconfig_req_raw(ctx->ep, sha256, sha256_len, bytes,
bytes_len);
if (rc < 0) {
return rc;
}
struct timespec _dl; cbuf_deadline_10s(&_dl);
rc = cbuf_timedget(&(ctx->wbuf), &status, &_dl);
if (rc < 0) {
return rc;
}
return status;
}
int xce_runtask(xce_ctx_t* ctx, xce_taskrun_t* runinfo)
{
int rc;

11
src/nixSupport/xce.nix
View file

@ -35,6 +35,17 @@ stdenv.mkDerivation {
struct2json
(capnpc.overrideAttrs (oldAttrs: {
separateDebugInfo = true;
# bug-918: upstream xbuild's capnpc package.nix pins src to the
# MUTABLE `main` ref of shen390s/c-capnproto with a hardcoded sha256,
# so it hard-breaks (fixed-output hash mismatch) every time that
# branch moves. Pin to an immutable commit here instead so this repo
# stops depending on upstream's moving target.
src = pkgs.fetchFromGitHub {
owner = "shen390s";
repo = "c-capnproto";
rev = "e45ff682c4756ac5067cbac6d0f282a8ea5f0155";
sha256 = "sha256-9rTlcTia+s9ARGeTi+8auGvM0RkS+wZK7aIkzecaeSk=";
};
}))
which
findutils

31
src/schema/controller.capnp
View file

@ -752,6 +752,30 @@ struct SnapshotHeader $C.mapname("capnp_snap_hdr_t") {
headerChecksum @8 :UInt32;
}
# ── Upgrade snapshot admin-state frames (config-upgrade-snapshot task 1,
# Req 1.9/5.7) ──────────────────────────────────────────────────────────
# Independently-extensible Cap'n Proto frames for the on-disk checkpoint's
# host/queue admin-state sections (SNAPSHOT_FORMAT_VERSION 2, persist.c).
# New admin-state fields (e.g. a lock reason or tag) get new ordinals under
# capnp's own schema-evolution rules -- no future SNAPSHOT_FORMAT_VERSION
# bump needed per field addition. Deliberately NOT mapname'd onto
# Worker/QueueConf (design.md Design Invariant 5): those live runtime
# structs bundle static attrs and dynamic state together, which a snapshot
# frame must never carry (Requirement 1.7 excludes dynamic fields). Each
# maps instead onto its own frame-only C counterpart in typemap.h, matching
# the existing capnp_snap_hdr_t/capnp_persist_cfg_t/capnp_repl_envelope_t
# convention already established below for schema-only wire concepts with
# no corresponding "live" C struct.
struct HostAdminState $C.mapname("capnp_host_admin_state_t") {
name @0: Text;
status @1: UInt32;
}
struct QueueAdminState $C.mapname("capnp_queue_admin_state_t") {
name @0: Text;
status @1: UInt32;
}
# ── HA heartbeat ────────────────────────────────────────────────────
struct HeartbeatMsg $C.mapname("heartbeat_msg_t") {
senderPriority @0: UInt32 $C.mapname("sender_priority");
@ -781,6 +805,13 @@ struct ReplConfigPushRequest $C.mapname("repl_config_push_req_t") {
struct ReplConfigPushReply $C.mapname("repl_config_push_reply_t") {
accepted @0: Bool $C.mapname("accepted");
versionTag @1: UInt64 $C.mapname("version_tag");
# raft-adapter-conformance Requirement 7.4: the replying member's current
# term, so a higher term can be discovered through this reply channel the
# same way it already is through heartbeat and WAL-NACK replies. Additive
# field (ordinal 2) — an old-code standby's reply decodes as term=0 on a
# new-code primary, which can never look like a higher term (terms start
# at 1), so no spurious adoption occurs under mixed-version rollout.
term @2: UInt64 $C.mapname("term");
}
struct UpgradeConfigRequest $C.mapname("upgrade_config_req_t") {

56
src/xceadmin/src/xceadmin/discovery.py
View file

@ -12,13 +12,30 @@ Note: design.md's Dependencies list for this component names only
``xcecommon.wire``, but decoding ``RequestReply`` to inspect the redirect
fields genuinely requires ``xcecommon.schema`` too -- treated as a design
doc omission, not a license to hand-roll capnp decoding here.
Reply envelope (task 6.1 bug fix, live-verified 2026-07-30 against a real
deployed controller): every reply ``xce_reply()`` (``src/lib/request.c``)
sends over the wire -- ``config push``'s included -- is wrapped in an
outer ``Request{opcode: OPCode_reply, body: union{replymsg @4:
RequestReply}}`` envelope (``controller.capnp:645-651``) before being
serialized, never a bare ``RequestReply``. Decoding raw reply bytes
directly as ``RequestReply`` (the previous behavior here and in
``xceadmin.push``) misaligns every pointer offset and crashes with
``capnp.KjException: ... non-list pointer where text was expected`` on the
very first field access. :func:`decode_reply` is the single place this
unwrap happens -- this module owns it (it already owns "understand the
``RequestReply`` wire shape"), and ``xceadmin.push`` imports it rather than
duplicating the unwrap at its own call site.
"""
from __future__ import annotations
import contextlib
import sys
from xcecommon.errors import ConnectionTimeoutError, NoPrimaryError, TooManyRedirectsError
import capnp
from xcecommon.errors import ConnectionTimeoutError, NoPrimaryError, ReplyDecodeError, TooManyRedirectsError
from xcecommon.schema import load_schema
#: "Follow at most 3 redirect hops before reporting an error" (3.2) -- up
@ -26,13 +43,46 @@ from xcecommon.schema import load_schema
_MAX_REDIRECT_HOPS = 3
@contextlib.contextmanager
def decode_reply(reply_bytes: bytes):
"""Decode ``reply_bytes`` as the ``Request{opcode: reply,
body.replymsg: RequestReply}`` envelope every controller reply is
wrapped in on the wire (module docstring), and yield the inner
``RequestReply`` for field access -- e.g. ``with decode_reply(b) as
reply: reply.status``.
Raises :class:`ReplyDecodeError` (Requirement 7.3) if ``reply_bytes``
cannot be decoded as a ``Request`` at all, or decodes but
``body.which()`` is not ``"replymsg"`` (the bytes are not actually a
reply envelope -- this is what the old, wrong bare-``RequestReply``
wire shape now produces: a defaulted ``"nomsg"`` variant instead of a
crash, confirmed empirically). pycapnp's malformed-message errors are
often raised lazily on field access rather than at ``from_bytes()``
time, and a byte length that isn't a multiple of 8 words raises a
plain ``ValueError`` instead of ``capnp.KjException`` -- both are
caught here (verified against every corrupt/truncated/empty/garbage
byte shape) so no raw capnp/decode exception ever escapes this
function.
"""
mod = load_schema()
try:
with mod.Request.from_bytes(reply_bytes) as request:
which = request.body.which()
if which != "replymsg":
reason = f"expected reply envelope (body.replymsg), got body.{which}"
raise ReplyDecodeError(f"invalid reply: {reason}", reason)
yield request.body.replymsg
except (capnp.KjException, ValueError) as exc:
reason = str(exc).strip() or "malformed capnp reply envelope"
raise ReplyDecodeError(f"invalid reply: {reason}", reason) from exc
def _redirect_target(reply_bytes: bytes) -> tuple[str, int] | None:
"""``None`` when ``reply_bytes`` carries no redirect (empty
``redirectHost`` or zero ``redirectPort``) -- both must be non-empty
for this to be a real redirect (3.1).
"""
mod = load_schema()
with mod.RequestReply.from_bytes(reply_bytes) as reply:
with decode_reply(reply_bytes) as reply:
host, port = reply.redirectHost, reply.redirectPort
if host and port:
return host, port

4
src/xceadmin/src/xceadmin/push.py
View file

@ -49,7 +49,7 @@ from pathlib import Path
from xcecommon import wire
from xcecommon.errors import ReplyDecodeError
from xcecommon.schema import load_schema
from xceadmin.discovery import follow_redirects
from xceadmin.discovery import decode_reply, follow_redirects
_DEFAULT_PORT = 9991
@ -113,7 +113,7 @@ def push_config(cluster_bin_path: str, controller: str) -> PushResult:
host, port = _split_controller(controller)
_, _, reply_bytes = follow_redirects(host, port, _send_via(request_bytes))
with mod.RequestReply.from_bytes(reply_bytes) as reply:
with decode_reply(reply_bytes) as reply:
if reply.status != 0:
return PushResult(
version_tag=0, acked_count=0, error=f"controller returned status {reply.status}"

99
src/xceadmin/tests/_live_e2e_common.py Normal file
View file

@ -0,0 +1,99 @@
"""Shared helpers for xceadmin's live E2E test suite (task 6.2's
``test_e2e_live_push.py`` and task 6.3's ``test_e2e_live_ha_push.py``) --
binary discovery, ephemeral port allocation, waiting for a listener to come
up, and the ``versionTag=.../ackedCount=...`` stdout parser every live
``config push`` invocation needs to check.
Extracted here (task 6.3) rather than duplicated a second time: task 6.3's
own text asks whether task 6.2's fixture/helpers can be reused for the
multi-node case rather than duplicated. These four helpers (binary
discovery, port allocation, port-wait, the versionTag/ackedCount regex) are
identical text in both files pre-extraction -- genuine repetition, not
speculative generality -- so they move here and both test files import from
it. The two files' *fixtures* (``live_controller`` vs. ``live_ha_cluster``)
stay separate: single-node vs. 3-node deploy/launch/election-wait/cleanup
sequences differ enough (peer strings, staggered multi-process start,
primary-election polling) that merging them would trade a real readability
loss for a smaller one.
Not a test module itself (no ``test_*`` functions/classes), so pytest's
default collection (``test_*.py``/``*_test.py``) leaves it alone; it is
picked up as an ordinary same-directory import instead (this project's
xceadmin/tests/ has no ``__init__.py``, so pytest's default "prepend"
import mode puts this directory on ``sys.path`` for every test module in
it -- the same mechanism that already lets every test file here import
``conftest.py``'s fixtures without a package-qualified path).
"""
from __future__ import annotations
import re
import socket
import time
from pathlib import Path
SRC_DIR = Path(__file__).resolve().parents[2]
REPO_CONFIGS_DIR = SRC_DIR / "configs"
DEPLOY_HA_SH = SRC_DIR / "tools" / "deploy-ha.sh"
# Mirrors deploy-ha.sh's own xceadmin/controller discovery order
# (src/tools/deploy-ha.sh) -- xce.build (asan, the default profile) first,
# then release, then debug.
CONTROLLER_CANDIDATES = [
SRC_DIR / "xce.build" / "controller" / "xce_controller",
SRC_DIR / "build.release" / "controller" / "xce_controller",
SRC_DIR / "build.dbg" / "controller" / "xce_controller",
]
XCEADMIN_CANDIDATES = [
SRC_DIR / "xce.build" / "xceadmin" / ".venv" / "bin" / "xceadmin",
SRC_DIR / "build.release" / "xceadmin" / ".venv" / "bin" / "xceadmin",
SRC_DIR / "build.dbg" / "xceadmin" / ".venv" / "bin" / "xceadmin",
]
READY_TIMEOUT_S = 20.0
VERSION_TAG_RE = re.compile(r"versionTag=(\d+)\s+ackedCount=(\d+)")
def find_first_executable(candidates: list[Path]) -> Path | None:
for candidate in candidates:
if candidate.is_file() and (candidate.stat().st_mode & 0o111):
return candidate
return None
def free_tcp_port() -> int:
"""A port that was free at bind-time -- the same best-effort ephemeral-
port trick used to avoid hardcoding a fixed port a test could collide
on. The bind-then-immediately-close TOCTOU window is acceptable for a
single-shot test deploy.
"""
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as probe:
probe.bind(("127.0.0.1", 0))
return probe.getsockname()[1]
def free_tcp_ports(count: int) -> list[int]:
"""``count`` distinct ports from :func:`free_tcp_port`, de-duplicated
across the batch (a multi-node deploy needs several at once; calling
:func:`free_tcp_port` in a bare loop can occasionally return the same
port twice since each call binds-then-immediately-releases before the
next one binds). Returned in ascending order for a stable, readable
ctrl-1/ctrl-2/ctrl-3 assignment.
"""
ports: set[int] = set()
while len(ports) < count:
ports.add(free_tcp_port())
return sorted(ports)
def wait_for_port(host: str, port: int, timeout_s: float) -> None:
deadline = time.monotonic() + timeout_s
last_err: OSError | None = None
while time.monotonic() < deadline:
try:
with socket.create_connection((host, port), timeout=1.0):
return
except OSError as exc:
last_err = exc
time.sleep(0.2)
raise TimeoutError(f"controller never opened {host}:{port}: {last_err}")

28
src/xceadmin/tests/conftest.py
View file

@ -9,6 +9,8 @@ from pathlib import Path
import pytest
from xcecommon.schema import load_schema
@dataclass(frozen=True)
class TlsCertPair:
@ -42,3 +44,29 @@ def tls_certs(tmp_path_factory) -> TlsCertPair:
capture_output=True,
)
return TlsCertPair(cert_file=cert_file, key_file=key_file)
@pytest.fixture
def wrap_reply():
"""Factory fixture: wrap a ``RequestReply`` message builder in the
outer ``Request{opcode: reply, body.replymsg: ...}`` envelope every
real controller reply is sent in on the wire (``xce_reply()``,
``src/lib/request.c``; ``controller.capnp:645-651``) -- returns the
serialized envelope bytes.
Task 6.1 bug fix: mock TLS servers across this suite previously sent a
bare ``RequestReply`` on the wire, which matched the old (wrong)
client-side decode assumption. ``xceadmin.discovery.decode_reply`` now
expects the real envelope shape, so every mock server reply must be
wrapped through this helper instead.
"""
def _wrap(reply_msg) -> bytes:
mod = load_schema()
request = mod.Request.new_message()
request.opcode = "reply"
request.seq = reply_msg.seq
request.body.replymsg = reply_msg
return request.to_bytes()
return _wrap

8
src/xceadmin/tests/test_cli.py
View file

@ -196,7 +196,7 @@ def cluster_bin(tmp_path):
return str(path)
def test_config_push_success_exits_zero(mock_controller, cluster_bin):
def test_config_push_success_exits_zero(mock_controller, cluster_bin, wrap_reply):
from xcecommon import wire
mod = load_schema()
@ -209,7 +209,7 @@ def test_config_push_success_exits_zero(mock_controller, cluster_bin):
upgrade_reply.versionTag = 5
upgrade_reply.ackedCount = 3
upgrade_reply.error = ""
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
srv = mock_controller(_handler)
@ -224,7 +224,7 @@ def test_config_push_success_exits_zero(mock_controller, cluster_bin):
assert "ackedCount=3" in result.output
def test_config_push_zero_acked_count_exits_nonzero(mock_controller, cluster_bin):
def test_config_push_zero_acked_count_exits_nonzero(mock_controller, cluster_bin, wrap_reply):
from xcecommon import wire
mod = load_schema()
@ -237,7 +237,7 @@ def test_config_push_zero_acked_count_exits_nonzero(mock_controller, cluster_bin
upgrade_reply.versionTag = 0
upgrade_reply.ackedCount = 0
upgrade_reply.error = "no quorum"
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
srv = mock_controller(_handler)

49
src/xceadmin/tests/test_config_push_mock.py
View file

@ -99,7 +99,7 @@ def cluster_bin(tmp_path):
# ---------------------------------------------------------------------------
def test_push_config_successful_commit(mock_controller, cluster_bin):
def test_push_config_successful_commit(mock_controller, cluster_bin, wrap_reply):
mod = load_schema()
received = {}
@ -117,7 +117,7 @@ def test_push_config_successful_commit(mock_controller, cluster_bin):
upgrade_reply.versionTag = 7
upgrade_reply.ackedCount = 3
upgrade_reply.error = ""
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
srv = mock_controller(_handler)
@ -133,7 +133,7 @@ def test_push_config_successful_commit(mock_controller, cluster_bin):
assert received["sha256"] == hashlib.sha256(Path(cluster_bin).read_bytes()).digest()
def test_push_config_follows_one_redirect_to_primary(mock_controller, cluster_bin):
def test_push_config_follows_one_redirect_to_primary(mock_controller, cluster_bin, wrap_reply):
mod = load_schema()
def _standby_handler(conn):
@ -144,7 +144,7 @@ def test_push_config_follows_one_redirect_to_primary(mock_controller, cluster_bi
# port filled in once the primary server exists, see below
reply.redirectPort = _standby_handler.primary_port
reply.data.nodata = None
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
def _primary_handler(conn):
wire.recv_frame(conn)
@ -154,7 +154,7 @@ def test_push_config_follows_one_redirect_to_primary(mock_controller, cluster_bi
upgrade_reply.versionTag = 9
upgrade_reply.ackedCount = 2
upgrade_reply.error = ""
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
primary_srv = mock_controller(_primary_handler)
_standby_handler.primary_port = primary_srv.port
@ -166,7 +166,7 @@ def test_push_config_follows_one_redirect_to_primary(mock_controller, cluster_bi
assert result.acked_count == 2
def test_push_config_non_zero_status_reports_error_not_exception(mock_controller, cluster_bin):
def test_push_config_non_zero_status_reports_error_not_exception(mock_controller, cluster_bin, wrap_reply):
mod = load_schema()
def _handler(conn):
@ -174,7 +174,7 @@ def test_push_config_non_zero_status_reports_error_not_exception(mock_controller
reply = mod.RequestReply.new_message()
reply.status = 1
reply.data.debugmsg = "simulated failure"
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
srv = mock_controller(_handler)
@ -184,7 +184,7 @@ def test_push_config_non_zero_status_reports_error_not_exception(mock_controller
assert "status 1" in result.error
def test_push_config_wrong_reply_variant_raises_reply_decode_error(mock_controller, cluster_bin):
def test_push_config_wrong_reply_variant_raises_reply_decode_error(mock_controller, cluster_bin, wrap_reply):
mod = load_schema()
def _handler(conn):
@ -192,7 +192,34 @@ def test_push_config_wrong_reply_variant_raises_reply_decode_error(mock_controll
reply = mod.RequestReply.new_message()
reply.status = 0
reply.data.nodata = None # not upgradeconfigreplymsg
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
srv = mock_controller(_handler)
with pytest.raises(ReplyDecodeError):
push.push_config(cluster_bin, f"127.0.0.1:{srv.port}")
def test_push_config_rejects_bare_requestreply_bytes_old_wire_shape_regression(
mock_controller, cluster_bin
):
"""Regression test (task 6.1): a mock server that still sends the OLD,
wrong shape -- a bare RequestReply with no outer Request envelope --
must be cleanly rejected as ReplyDecodeError, not silently accepted
and not an unhandled capnp.KjException. This locks in that the fixed
client only accepts the real controller wire shape.
"""
mod = load_schema()
def _handler(conn):
wire.recv_frame(conn)
reply = mod.RequestReply.new_message()
reply.status = 0
upgrade_reply = reply.data.init("upgradeconfigreplymsg")
upgrade_reply.versionTag = 1
upgrade_reply.ackedCount = 1
upgrade_reply.error = ""
wire.send_frame(conn, reply.to_bytes()) # NOT wrapped -- old shape
srv = mock_controller(_handler)
@ -301,7 +328,7 @@ def test_probe_config_has_no_wire_level_rejection_signal_by_design(
# ---------------------------------------------------------------------------
def test_push_config_propagates_too_many_redirects_end_to_end(mock_controller, cluster_bin):
def test_push_config_propagates_too_many_redirects_end_to_end(mock_controller, cluster_bin, wrap_reply):
"""A real chain of 4 redirecting mock servers -- push_config's own
composition with xceadmin.discovery must surface TooManyRedirectsError,
not silently follow forever or return a bogus result.
@ -317,7 +344,7 @@ def test_push_config_propagates_too_many_redirects_end_to_end(mock_controller, c
reply.redirectHost = "127.0.0.1"
reply.redirectPort = next_port_getter()
reply.data.nodata = None
wire.send_frame(conn, reply.to_bytes())
wire.send_frame(conn, wrap_reply(reply))
return _handler

92
src/xceadmin/tests/test_discovery.py
View file

@ -1,24 +1,35 @@
"""Unit tests for ``xceadmin.discovery`` (design.md, ``xceadmin.discovery``;
Requirements 3.1, 3.2, 3.3, 3.4).
Requirements 3.1, 3.2, 3.3, 3.4, 7.3).
"""
from __future__ import annotations
import pytest
from xcecommon.errors import ConnectionTimeoutError, NoPrimaryError, TooManyRedirectsError
from xcecommon.errors import ConnectionTimeoutError, NoPrimaryError, ReplyDecodeError, TooManyRedirectsError
from xcecommon.schema import load_schema
from xceadmin import discovery
def _reply_bytes(redirect_host: str = "", redirect_port: int = 0) -> bytes:
"""Builds the real ``Request{opcode: reply, body.replymsg:
RequestReply}`` envelope -- the actual wire shape every controller
reply is sent in (``xce_reply()``, ``src/lib/request.c``;
``controller.capnp:645-651``; task 6.1). NOT a bare ``RequestReply`` --
see ``test_redirect_target_rejects_bare_requestreply_bytes_old_wire_shape``
below for what happens when the old, wrong shape is decoded.
"""
mod = load_schema()
msg = mod.RequestReply.new_message()
inner = mod.RequestReply.new_message()
if redirect_host:
msg.redirectHost = redirect_host
inner.redirectHost = redirect_host
if redirect_port:
msg.redirectPort = redirect_port
return msg.to_bytes()
inner.redirectPort = redirect_port
request = mod.Request.new_message()
request.opcode = "reply"
request.seq = 1
request.body.replymsg = inner
return request.to_bytes()
def test_follow_redirects_no_redirect_returns_first_reply_unchanged():
@ -138,3 +149,72 @@ def test_follow_redirects_zero_port_is_not_treated_as_a_redirect():
assert (final_host, final_port) == ("node1", 9991)
assert reply_bytes == reply
# ---------------------------------------------------------------------------
# decode_reply(): Request{body.replymsg} envelope unwrap (task 6.1)
# ---------------------------------------------------------------------------
def test_decode_reply_unwraps_envelope_and_yields_request_reply():
"""The real wire shape -- Request{opcode: reply, body.replymsg:
RequestReply} -- decodes cleanly and yields the inner RequestReply.
"""
reply_bytes = _reply_bytes(redirect_host="primary", redirect_port=9991)
with discovery.decode_reply(reply_bytes) as reply:
assert reply.redirectHost == "primary"
assert reply.redirectPort == 9991
def test_decode_reply_rejects_bare_requestreply_bytes_old_wire_shape():
"""Regression test (task 6.1): the OLD, wrong client assumption was
that a reply is a bare RequestReply with no outer Request envelope.
Feeding decode_reply exactly that (unwrapped) byte shape must not
silently succeed and must not raise a raw capnp exception -- it must
be cleanly rejected as ReplyDecodeError, proving this is no longer
accepted as a valid reply shape.
"""
mod = load_schema()
bare_bytes = mod.RequestReply.new_message().to_bytes()
with pytest.raises(ReplyDecodeError) as exc_info:
with discovery.decode_reply(bare_bytes) as _reply:
pass
assert "replymsg" in exc_info.value.message
def test_decode_reply_wraps_truncated_bytes_as_reply_decode_error():
"""A genuinely corrupt/truncated reply must surface as
ReplyDecodeError (Requirement 7.3), never an unhandled
capnp.KjException or ValueError.
"""
full = _reply_bytes(redirect_host="primary", redirect_port=9991)
truncated = full[: len(full) // 2]
with pytest.raises(ReplyDecodeError):
with discovery.decode_reply(truncated) as _reply:
pass
def test_decode_reply_wraps_non_capnp_bytes_as_reply_decode_error():
"""Bytes that aren't capnp at all (e.g. a length not a multiple of 8
words) raise a plain ValueError inside pycapnp -- confirm that is also
converted to ReplyDecodeError, not left to escape raw.
"""
with pytest.raises(ReplyDecodeError):
with discovery.decode_reply(b"not a capnp message at all") as _reply:
pass
def test_follow_redirects_propagates_reply_decode_error_on_corrupt_reply():
"""follow_redirects' own composition with decode_reply (via
_redirect_target) must surface the same ReplyDecodeError, not crash.
"""
def send_fn(host, port):
return b"\x00" * 4 # too short to be a valid capnp frame
with pytest.raises(ReplyDecodeError):
discovery.follow_redirects("node1", 9991, send_fn)

496
src/xceadmin/tests/test_e2e_live_ha_push.py Normal file
View file

@ -0,0 +1,496 @@
"""Live 3-node HA E2E coverage for `config push` -- success path + the
standby-contact case (task 6.3; Requirements 2.1, 2.5, 2.6, 3.1, 3.2, 3.3).
Task 6.2's ``test_e2e_live_push.py`` deploys a genuinely single-node
controller (zero HA peers), which structurally cannot exercise a real
positive ``ackedCount`` / exit-0 success path (Requirement 2.6) -- a
peer-less controller has no standbys to ACK, so ``ackedCount`` is always 0
there (see task 6.2's Implementation Notes entry and that file's own module
docstring). This file closes THAT gap against one live 3-node
``xce_controller`` cluster (reusing ``deploy-ha.sh``'s existing multi-node
pattern -- see ``live_ha_cluster`` below):
``test_live_config_push_against_primary_reaches_success_path`` genuinely
exits 0 with a positive ``ackedCount`` from real standby ACKs.
REAL FINDING, not a test bug -- the redirect scenario does NOT close the
way the task text originally anticipated: pushing `config push` to a
*standby* does NOT redirect to the primary against a real controller
today, and this is not something a test-only change can fix. This is an
ALREADY-DOCUMENTED, pre-existing gap in design.md's own "Boundary
Commitments" section (`.kiro/specs/admin-config-tool/design.md`, "Out of
Boundary" -- "Redirect wiring for `OP_upgradeConfig`") and "Revalidation
Triggers" (`fill_redirect_reply()` extended to `OPCode_upgradeConfig` is
listed there as a FUTURE, not-yet-done trigger): `fill_redirect_reply()`
(`join.c:27-31`) is only ever invoked from the `OPCode_join` path
(`join.c:277`); nothing calls it for `OPCode_upgradeConfig`.
`RequestReply.redirectHost`/`redirectPort` are therefore never populated
for an `upgradeConfig` reply. Confirmed directly in `cluster.c`'s
`myreq_handler`: a standby's `OP_upgradeConfig` request is rejected by the
generic `ha_dispatch_gate()` role check (`ha.c`, opcode absent from the
`CTRL_STANDBY` allow-list) BEFORE the opcode-specific `case
OPCode_upgradeConfig:` handler ever runs -- the gate-rejection reply path
(`cluster.c` ~line 273-280) sends `reply.status = -XCE_ERR_STANDBY`
(`errnox.h`: `E_XCE_STANDBY == XCE_ERRNO_BASE + 13 == 10014`) with
`data_which = RequestReply_data_nodata` and never touches
`redirectHost`/`redirectPort` at all. Live-verified against a real 3-node
cluster: pushing to a standby returns `versionTag=0 ackedCount=0`, exit 1,
stderr `"Error: controller returned status -10014"` -- no redirect line,
no success. `xceadmin.discovery.follow_redirects`/`decode_reply` behave
exactly per Requirement 3.1's contract given this reply (both fields
absent/empty => no redirect => the bare rejection is returned as-is); the
gap is entirely server-side and out of this task's test-only boundary to
fix (design.md: "This is a controller defect, not an xceadmin defect;
xceadmin implements the client side of the documented contract... and
cannot itself fix the gate"). Task 5.6 already covers the SUCCESSFUL
redirect-follow path against mocks (`test_config_push_mock.py`), which is
still the only place that path is exercised.
``test_live_config_push_against_standby_returns_standby_rejection_not_redirect``
below live-verifies and regression-locks this REAL current behavior
instead of the hypothetical "redirect then succeed" outcome -- the same
resolution task 6.2 used for its own real-vs-literal-task-text mismatch
(single-node `ackedCount == 0`/exit 1). The moment a future controller-side
task wires `fill_redirect_reply()` for `OPCode_upgradeConfig` (already
tracked as a Revalidation Trigger in design.md), THIS test will start
failing on the "no redirect line" assertion -- that failure is the signal
to replace it with a real success-path assertion, matching what this
docstring originally described before investigation.
Shares binary discovery / port / stdout-parsing helpers with
``test_e2e_live_push.py`` via ``_live_e2e_common`` (extracted there as part
of this task -- see that module's docstring for the extraction rationale).
The multi-node deploy/launch/election-wait/cleanup fixture itself
(``live_ha_cluster``) is NOT shared with task 6.2's ``live_controller``:
staggered 3-process start, per-peer log polling for primary election, and
building the ``ctrl-N=host:port`` peers string for 3 nodes are enough of a
different shape from a single ``deploy-ha.sh`` call + single process launch
that merging them would cost more readability than it would save.
Skip convention: identical to task 6.2 -- an imperative ``pytest.skip``
inside the fixture (no controller binary / no ``openssl``), plus an
in-test skip for a missing ``xceadmin`` binary, both matching this suite's
existing cert-dependent skip precedent (task 5.3). Skipping is the NORMAL
outcome under the fast ``xce-build.sh xceadmin`` / meson path, which runs
pytest from an isolated ``repo-mirror`` with no sibling
``xce.build``/``build.dbg``/``build.release`` controller binaries.
Process-leak fix (task 6.2's own finding, reused here for all 3 nodes):
each controller is launched as the real ``xce_controller`` binary
directly, never through ``deploy-ha.sh``'s generated ``scripts/start.sh``
bash wrapper -- that wrapper runs the binary as bash's non-``exec``'d
child, so ``Popen.terminate()`` on the wrapper does not reach the real
controller process and orphans it.
Election-wait (round-2 remediation -- round-1 review REJECTED the
single-phase version of this gate as flaky, ~75% failure rate over 8 live
runs): polls each of the 3 controller log files for the literal substring
``"CTRL_PRIMARY"`` (``ha.c``'s own election-complete log lines, e.g. ``"HA
FSM: -> CTRL_PRIMARY (term=...)"`` / ``"HA FSM (raft): -> CTRL_PRIMARY
(term=...)"``) rather than a fixed sleep -- the same convention this
repo's own ``test-heartbeat-tolerance.sh`` (``wait_for_primary()``)
already uses, generalized here to check all 3 logs (that script only
checks ``ctrl-1.log``, relying on ctrl-1 usually winning the first
election since it starts first; this test does not assume which node
wins). Round 1 stopped there; that is what made it flaky.
This file now ALSO ports the second half of ``test-heartbeat-tolerance.sh``'s
own readiness gate that round 1 skipped: that script never proceeds past
``wait_for_primary()`` without immediately also calling
``wait_for_standbys()`` (confirming both other nodes' logs show
``CTRL_STANDBY``) -- round 1's fixture treated a single ``CTRL_PRIMARY``
log line, alone, as "cluster ready," which does not prove the cluster has
stopped electing: this repo's 3-node bootstrap starts all 3 nodes
``CTRL_STANDBY`` and self-promotes via a PreVote-triggered failover after a
uniform ~10.2s liveness timeout (``ha.h``: ``HA_DEFAULT_HEARTBEAT_TIMEOUT_MS
= 10000``) against only a 0.5s start stagger -- a genuine 3-way race where a
second, later election can reshuffle who is primary between "detected
primary" and "pushed to it." ``_wait_for_standbys`` below is that missing
half, generalized the same way ``_wait_for_primary`` already was (checking
the two specific non-primary nodes by name, not a fixed ctrl-2/ctrl-3
assumption).
Live reproduction during round-2 remediation (a standalone repro harness,
4 attempts, all with near-identical timing: ports open ~1.2s, CTRL_PRIMARY
~10.2s) surfaced a THIRD failure mode neither round 1 nor the plain
standby-wait alone fully closes: even after both markers are confirmed,
the freshly-promoted primary's own admin port can still be transiently
``ECONNREFUSED`` or accept the TCP connection but reply with an internal
``ETIMEDOUT`` (status -110) for a short window immediately after the
``CTRL_PRIMARY`` log line is written -- consistent with post-promotion
socket/RPC-pipeline warmup that the log line itself does not wait for.
``_POST_ELECTION_SETTLE_S`` (one full ``heartbeat_interval_ms``, ``ha.h``:
``HA_DEFAULT_HEARTBEAT_INTERVAL_MS = 3000`` -- the same constant
``test-heartbeat-tolerance.sh`` keys its own post-election settle sleeps
to) plus a fresh ``wait_for_port`` re-probe of the primary's address right
before the fixture yields (reusing the exact same helper used at initial
boot, not a new one) closes that window: a still-flapping port keeps
retrying inside ``wait_for_port`` instead of racing straight into the
push and surfacing as a confusing ``ConnectionRefusedError``/``ETIMEDOUT``
deep inside the test body.
"""
from __future__ import annotations
import re
import shutil
import subprocess
import time
from dataclasses import dataclass
from pathlib import Path
import pytest
from _live_e2e_common import CONTROLLER_CANDIDATES
from _live_e2e_common import DEPLOY_HA_SH, REPO_CONFIGS_DIR
from _live_e2e_common import READY_TIMEOUT_S
from _live_e2e_common import VERSION_TAG_RE
from _live_e2e_common import XCEADMIN_CANDIDATES
from _live_e2e_common import find_first_executable
from _live_e2e_common import free_tcp_ports
from _live_e2e_common import wait_for_port
_N_NODES = 3
_STAGGER_S = 0.5
_ELECTION_TIMEOUT_S = 30.0
_ELECTION_POLL_S = 1.0
# One full heartbeat_interval_ms (ha.h: HA_DEFAULT_HEARTBEAT_INTERVAL_MS =
# 3000, the same constant test-heartbeat-tolerance.sh's own post-election
# settle sleeps are keyed to) -- see module docstring's "Election-wait"
# section for the live-reproduced failure mode this closes.
_POST_ELECTION_SETTLE_S = 3.0
_PRIMARY_MARKER = "CTRL_PRIMARY"
_STANDBY_MARKER = "CTRL_STANDBY"
# Primary-side evidence that at least one standby genuinely ACKed the push
# (config_repl.c's push_config_to_standbys, on majority ACK):
# "Config committed on majority (tag=..., pushed=X/Y standbys)".
_MAJORITY_ACK_RE = re.compile(
r"Config committed on majority \(tag=([0-9a-f]{16}), pushed=(\d+)/(\d+) standbys\)"
)
# Standby-side evidence that IT received and saved the pushed bytes
# (config_repl.c's handle_repl_config_push, on successful conf_save_config):
# "handle_repl_config_push: config bytes saved out-of-band (pushed tag=...)".
_STANDBY_SAVED_RE = re.compile(
r"handle_repl_config_push: config bytes saved out-of-band \(pushed tag=([0-9a-f]{16})\)"
)
@dataclass(frozen=True)
class HaNode:
name: str
addr: str
log_file: Path
@dataclass(frozen=True)
class HaCluster:
primary: HaNode
standbys: list[HaNode]
def _wait_for_primary(nodes: list[HaNode], timeout_s: float) -> HaNode:
"""Poll every node's log for ``ha.c``'s own election-complete marker
(module docstring) rather than a fixed sleep. Returns whichever node's
log shows it first; raises ``TimeoutError`` if none does within
``timeout_s``.
"""
deadline = time.monotonic() + timeout_s
while time.monotonic() < deadline:
for node in nodes:
if node.log_file.is_file() and _PRIMARY_MARKER in node.log_file.read_text():
return node
time.sleep(_ELECTION_POLL_S)
raise TimeoutError(
f"no node reached {_PRIMARY_MARKER} within {timeout_s}s "
f"(logs: {[str(n.log_file) for n in nodes]})"
)
def _wait_for_standbys(standby_nodes: list[HaNode], timeout_s: float) -> None:
"""Poll each of the (exactly two) non-primary nodes' own logs for
``ha.c``'s ``CTRL_STANDBY`` marker, mirroring
``test-heartbeat-tolerance.sh``'s ``wait_for_standbys()`` (module
docstring) -- that script never treats a cluster as ready on
``wait_for_primary()`` alone. Confirms BOTH other nodes have settled
into steady-state ``CTRL_STANDBY``, closing the round-1 race where a
single ``CTRL_PRIMARY`` log line was treated as sufficient even though
a second, later election could still be in flight. Raises
``TimeoutError`` naming whichever node(s) never got there.
"""
deadline = time.monotonic() + timeout_s
pending = {node.name: node for node in standby_nodes}
while time.monotonic() < deadline and pending:
for name, node in list(pending.items()):
if node.log_file.is_file() and _STANDBY_MARKER in node.log_file.read_text():
del pending[name]
if pending:
time.sleep(_ELECTION_POLL_S)
if pending:
raise TimeoutError(
f"standby node(s) {sorted(pending)} never reached {_STANDBY_MARKER} "
f"within {timeout_s}s "
f"(logs: {[str(n.log_file) for n in pending.values()]})"
)
@pytest.fixture
def live_ha_cluster(tmp_path: Path):
"""Deploy + start a real 3-node ``xce_controller`` HA cluster via the
project's own ``deploy-ha.sh``, wait for a primary to be elected, and
yield an :class:`HaCluster` naming which node is primary and which two
are standbys. Skips cleanly if no controller binary or ``openssl`` is
available (matching task 6.2's convention). Always kills all 3
controller processes and removes the scratch deploy directory, on both
success and failure paths.
"""
controller_bin = find_first_executable(CONTROLLER_CANDIDATES)
if controller_bin is None:
pytest.skip(
"no xce_controller binary found under xce.build/build.release/"
"build.dbg -- run xce-build.sh controller [profile] first to "
"exercise this live E2E test"
)
if shutil.which("openssl") is None:
pytest.skip("openssl not found on PATH -- required by deploy-ha.sh")
deploy_dir = tmp_path / "deploy"
ports = free_tcp_ports(_N_NODES)
bare_peers = ",".join(f"localhost:{p}" for p in ports)
deploy = subprocess.run(
["bash", str(DEPLOY_HA_SH), str(controller_bin), str(deploy_dir), bare_peers],
capture_output=True,
text=True,
)
assert deploy.returncode == 0, f"deploy-ha.sh failed: {deploy.stdout}\n{deploy.stderr}"
names = [f"ctrl-{i + 1}" for i in range(_N_NODES)]
addrs = [f"localhost:{p}" for p in ports]
# Matches deploy-ha.sh's own internal PEERS_STR construction exactly
# (src/tools/deploy-ha.sh) -- rebuilt here because this fixture launches
# each controller binary directly rather than through the generated
# scripts/start.sh wrapper (module docstring: process-leak fix).
peers_str = ",".join(f"{name}={addr}" for name, addr in zip(names, addrs))
procs: list[subprocess.Popen] = []
nodes: list[HaNode] = []
try:
for i, (name, addr) in enumerate(zip(names, addrs)):
log_file = deploy_dir / "logs" / f"{name}.log"
proc = subprocess.Popen(
[
str(controller_bin),
"--myname", name,
"--peers", peers_str,
"--workdir", str(deploy_dir / "controller"),
"--certdir", str(deploy_dir / "certs"),
"--logfile", str(log_file),
"--vcpus", "4",
],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
procs.append(proc)
nodes.append(HaNode(name=name, addr=addr, log_file=log_file))
if i < _N_NODES - 1:
time.sleep(_STAGGER_S)
for node, port in zip(nodes, ports):
wait_for_port("localhost", port, READY_TIMEOUT_S)
primary_node = _wait_for_primary(nodes, _ELECTION_TIMEOUT_S)
standby_nodes = [n for n in nodes if n.name != primary_node.name]
_wait_for_standbys(standby_nodes, _ELECTION_TIMEOUT_S)
# Settle window + fresh port reachability recheck (module
# docstring's "Election-wait" section has the full live-repro
# rationale): give the freshly-promoted primary one heartbeat
# interval to finish post-promotion warmup, then re-probe its port
# with the same helper used at initial boot rather than racing
# straight into the push against a still-flapping listener.
time.sleep(_POST_ELECTION_SETTLE_S)
primary_host, primary_port_str = primary_node.addr.rsplit(":", 1)
wait_for_port(primary_host, int(primary_port_str), READY_TIMEOUT_S)
yield HaCluster(primary=primary_node, standbys=standby_nodes)
finally:
for proc in procs:
proc.terminate()
for proc in procs:
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=10)
shutil.rmtree(deploy_dir, ignore_errors=True)
def _build_cluster_bin(xceadmin_bin: Path, tmp_path: Path) -> Path:
cluster_bin = tmp_path / "cluster.bin"
build_result = subprocess.run(
[str(xceadmin_bin), "config", "build", str(REPO_CONFIGS_DIR), "-o", str(cluster_bin)],
capture_output=True,
text=True,
)
assert build_result.returncode == 0, build_result.stderr
assert cluster_bin.is_file()
return cluster_bin
def _assert_no_decode_crash(push_result: subprocess.CompletedProcess) -> None:
# Task 6.1's regression lock, same as task 6.2 -- no raw capnp
# exception or unhandled traceback against REAL wire bytes.
assert "KjException" not in push_result.stderr, push_result.stderr
assert "Traceback" not in push_result.stderr, push_result.stderr
def _assert_primary_committed(log_text: str, version_tag_hex: str) -> None:
assert f"tag={version_tag_hex}" in log_text, log_text
assert "Config change committed" in log_text
assert f"Config upgraded to tag={version_tag_hex}" in log_text
assert f"config activated from committed marker tag={version_tag_hex}" in log_text
def test_live_config_push_against_primary_reaches_success_path(
live_ha_cluster: HaCluster, tmp_path: Path
) -> None:
"""Pushing directly to the elected primary of a real 3-node cluster
exits 0 with a genuinely positive ``ackedCount`` (Requirements 2.1,
2.5, 2.6) -- the exact success path task 6.2's single-node topology
cannot reach (no standbys there to ACK). Independently confirms via
log evidence: the primary's own commit/activate lines (same as task
6.2) PLUS the primary's majority-ACK barrier log line naming a
positive per-standby pushed count, AND at least one standby's own log
showing it received and saved the pushed bytes.
"""
xceadmin_bin = find_first_executable(XCEADMIN_CANDIDATES)
if xceadmin_bin is None:
pytest.skip("no xceadmin binary found under xce.build/build.release/build.dbg")
cluster_bin = _build_cluster_bin(xceadmin_bin, tmp_path)
push_result = subprocess.run(
[str(xceadmin_bin), "config", "push", str(cluster_bin), "--controller", live_ha_cluster.primary.addr],
capture_output=True,
text=True,
)
_assert_no_decode_crash(push_result)
match = VERSION_TAG_RE.search(push_result.stdout)
assert match, f"no versionTag/ackedCount found in stdout: {push_result.stdout!r}"
version_tag, acked_count = int(match.group(1)), int(match.group(2))
assert version_tag > 0
# The real, live 3-node success path task 6.2 could not reach: a
# genuine positive ackedCount (real standby ACKs exist now) and exit 0
# (Requirement 2.6's postcondition, cli.py's `acked_count > 0` check).
assert acked_count > 0, (
f"expected a positive ackedCount from real standby ACKs, got {acked_count} "
f"(stdout={push_result.stdout!r} stderr={push_result.stderr!r})"
)
assert push_result.returncode == 0, push_result.stderr
version_tag_hex = f"{version_tag:016x}"
primary_log_text = live_ha_cluster.primary.log_file.read_text()
_assert_primary_committed(primary_log_text, version_tag_hex)
# Majority-ACK barrier evidence on the primary's own log: at least one
# standby genuinely acked this exact tag (push_config_to_standbys).
majority_match = _MAJORITY_ACK_RE.search(primary_log_text)
assert majority_match, primary_log_text
assert majority_match.group(1) == version_tag_hex
pushed, total = int(majority_match.group(2)), int(majority_match.group(3))
assert pushed >= 1, f"expected >=1 standby acked, got pushed={pushed}/{total}"
# Independent standby-side evidence: at least one of the two standbys'
# own logs shows it actually received and saved these exact bytes
# (handle_repl_config_push), not just the primary's report of it.
#
# Each standby receives >1 push over the fixture's lifetime -- an
# automatic bootstrap sync of the primary's already-loaded config fires
# right after election (ha_dispatch_repl_action_for_standby), BEFORE
# this test's own xceadmin push -- so this must check every occurrence
# (re.finditer), not just the first (re.search would silently match the
# earlier bootstrap push's tag instead of this push's tag and always
# fail here, independent of whether the real ACK happened).
standby_saw_it = any(
m.group(1) == version_tag_hex
for standby in live_ha_cluster.standbys
for m in _STANDBY_SAVED_RE.finditer(standby.log_file.read_text())
)
assert standby_saw_it, [s.log_file.read_text() for s in live_ha_cluster.standbys]
# errnox.h: XCE_ERRNO_BASE=10001, E_XCE_STANDBY is the 14th entry (0-indexed
# 13) in that enum => 10001+13=10014. "Controller in CTRL_STANDBY; op is
# primary-only" -- exactly the rejection a standby's OP_upgradeConfig hits
# (module docstring). Named here instead of a bare magic number per this
# project's error-handling convention of surfacing XCE's own error
# vocabulary rather than an unexplained literal.
_E_XCE_STANDBY = 10014
def test_live_config_push_against_standby_returns_standby_rejection_not_redirect(
live_ha_cluster: HaCluster, tmp_path: Path
) -> None:
"""Pushing `config push` directly to a standby node against a REAL
controller today does NOT redirect to the primary -- it is flatly
rejected with ``E_XCE_STANDBY`` (module docstring: a pre-existing,
already-documented controller-side gap, `fill_redirect_reply()` never
wired for `OPCode_upgradeConfig`; out of this test-only task's
boundary to fix). This test live-verifies and regression-locks that
REAL current behavior: no redirect line, no success, and specifically
the `E_XCE_STANDBY` status rather than some other/unexpected failure
mode, so a future change to the rejection path itself would also be
caught here even before the redirect gap is closed.
Confirms via ``discovery.py``'s own stderr line (`follow_redirects`:
"redirected: host:port -> host:port") being ABSENT -- direct evidence
no redirect was attempted, not an inference from the push merely
failing for some other reason.
"""
xceadmin_bin = find_first_executable(XCEADMIN_CANDIDATES)
if xceadmin_bin is None:
pytest.skip("no xceadmin binary found under xce.build/build.release/build.dbg")
cluster_bin = _build_cluster_bin(xceadmin_bin, tmp_path)
standby_addr = live_ha_cluster.standbys[0].addr
primary_addr = live_ha_cluster.primary.addr
push_result = subprocess.run(
[str(xceadmin_bin), "config", "push", str(cluster_bin), "--controller", standby_addr],
capture_output=True,
text=True,
)
_assert_no_decode_crash(push_result)
# No redirect was attempted -- the standby's reply never carries
# redirectHost/redirectPort for this opcode (module docstring), so
# discovery.py's follow_redirects has nothing to follow and never
# prints its "redirected: ..." line.
unexpected_redirect_line = f"redirected: {standby_addr} -> {primary_addr}"
assert unexpected_redirect_line not in push_result.stderr, (
"a redirect line appeared -- if the controller-side redirect gap "
"documented in this file's module docstring (design.md 'Boundary "
"Commitments' / 'Revalidation Triggers') has been closed, this "
"test is stale and should be replaced with a real success-path "
f"assertion: {push_result.stderr!r}"
)
match = VERSION_TAG_RE.search(push_result.stdout)
assert match, f"no versionTag/ackedCount found in stdout: {push_result.stdout!r}"
version_tag, acked_count = int(match.group(1)), int(match.group(2))
# Real, live-verified standby-contact behavior today: the reply is a
# bare gate rejection (versionTag=0 ackedCount=0), not a redirect or a
# partial success.
assert version_tag == 0
assert acked_count == 0
assert push_result.returncode == 1
assert f"controller returned status -{_E_XCE_STANDBY}" in push_result.stderr, push_result.stderr
# The rejected push never reached the primary at all -- its log shows
# no upgrade activity from this attempt (independent confirmation this
# is a clean gate rejection, not a landed-then-lost push).
primary_log_text = live_ha_cluster.primary.log_file.read_text()
assert "Config marker failed to reach committed prefix" not in primary_log_text

200
src/xceadmin/tests/test_e2e_live_push.py Normal file
View file

@ -0,0 +1,200 @@
"""Live single-node E2E coverage for `config push` (task 6.2; Requirements
2.1, 2.5, 2.6, 3.1).
Every other push test in this suite (``test_config_push_mock.py``) talks to
a mock TLS server standing in for the controller. This file is the one
place the real ``xce_controller`` binary and the real ``xceadmin`` CLI are
exercised together end to end over the actual TLS+capnp wire protocol --
no mock anywhere. It exists specifically to regression-lock task 6.1's
reply-envelope decode fix (``xceadmin.discovery.decode_reply``) against a
REAL controller reply: task 6.1 itself discovered that this suite's mock
fixtures sent the wrong envelope shape and would never have caught the
crash that only showed up against a live controller.
Skip convention: an imperative ``pytest.skip`` inside the fixture, mirroring
``conftest.py``'s ``tls_certs`` openssl-missing skip (task 5.3's precedent)
-- not a module-level ``skipif``, since the discovery walk has to happen at
fixture-setup time. Skipping is the NORMAL outcome when this suite runs via
its usual path, ``xce-build.sh xceadmin``: that meson target builds only the
Python package (never the C controller) and runs pytest from an isolated
``repo-mirror`` copy of the source tree with no sibling
``xce.build``/``build.dbg``/``build.release`` controller binaries to find.
This test only runs for real when invoked directly against a real repo
checkout that has already built a controller (``xce-build.sh controller
[profile]``) -- the same "slow, on-demand live E2E" split this project
already uses for ``deploy-ha.sh``/``run-ha-soak.sh``/the ``xce-test-ha``
skill, which are likewise not part of the fast ``meson test`` path.
REAL finding, not a test bug (see the task's own CONCERNS writeup): a
genuinely single-node deployment (no HA peers -- ``deploy-ha.sh``'s
single-entry peer-list mode) has ZERO standbys to ACK, so
``push_config_to_standbys`` always returns ``acked=0`` even when the
config-change marker commits and activates successfully server-side
(verified live: the controller log shows "Config change committed",
"Config upgraded to tag=...", and "config activated from committed marker
tag=..." on every push). ``xceadmin.cli``'s postcondition (``acked_count
<= 0`` => ``ClickException``, Requirements 2.6/2.7) cannot distinguish "no
standbys exist" from "push genuinely failed", so a real single-node
``config push`` ALWAYS exits 1 with "push rejected (ackedCount == 0)" --
this is real, current, verified ``cli.py``/``push.py`` behavior against a
real controller, not a mock artifact or a test mistake. This test asserts
that REAL behavior (exit 1, ackedCount == 0) instead of a hypothetical
"exit 0", and separately verifies via the controller's own log file that
the push, commit, and activation all genuinely succeeded server-side --
the actual signal this test cares about (task 6.1's crash regression
lock), independent of the CLI's single-node-blind success heuristic (a
pre-existing gap in ``cli.py``/``push.py``, out of this task's boundary to
fix).
"""
from __future__ import annotations
import shutil
import subprocess
from pathlib import Path
import pytest
from _live_e2e_common import CONTROLLER_CANDIDATES as _CONTROLLER_CANDIDATES
from _live_e2e_common import DEPLOY_HA_SH, REPO_CONFIGS_DIR
from _live_e2e_common import READY_TIMEOUT_S as _READY_TIMEOUT_S
from _live_e2e_common import VERSION_TAG_RE as _VERSION_TAG_RE
from _live_e2e_common import XCEADMIN_CANDIDATES as _XCEADMIN_CANDIDATES
from _live_e2e_common import find_first_executable as _find_first_executable
from _live_e2e_common import free_tcp_port as _free_tcp_port
from _live_e2e_common import wait_for_port as _wait_for_port
@pytest.fixture
def live_controller(tmp_path: Path):
"""Deploy + start one real, single-node ``xce_controller`` via the
project's own ``deploy-ha.sh`` (module docstring: a single-entry peer
list, so zero standbys -- not "no HA machinery at all"). Skips cleanly
if no controller binary or ``openssl`` is available. Always kills the
controller process and removes the scratch deploy directory, on both
success and failure.
Yields ``(host_port_str, log_file_path)``.
Deliberately does NOT launch the controller through deploy-ha.sh's
generated ``scripts/start.sh``: that script runs ``$BINARY ...`` as a
plain (non-``exec``'d) simple command, so the controller is bash's
CHILD, not bash itself. ``Popen.terminate()`` only signals the bash
wrapper; a non-interactive bash's default SIGTERM handling exits
immediately without waiting on/forwarding to its child, orphaning a
live ``xce_controller`` process on every test run (caught empirically
-- four orphaned controllers accumulated across four manual runs
during this task's own verification, still bound to their ports).
Launching the binary directly, with the exact args ``start.sh`` would
have used, makes ``proc`` the real controller process so
``terminate()``/``kill()`` target it directly.
"""
controller_bin = _find_first_executable(_CONTROLLER_CANDIDATES)
if controller_bin is None:
pytest.skip(
"no xce_controller binary found under xce.build/build.release/"
"build.dbg -- run xce-build.sh controller [profile] first to "
"exercise this live E2E test"
)
if shutil.which("openssl") is None:
pytest.skip("openssl not found on PATH -- required by deploy-ha.sh")
deploy_dir = tmp_path / "deploy"
port = _free_tcp_port()
deploy = subprocess.run(
["bash", str(DEPLOY_HA_SH), str(controller_bin), str(deploy_dir), f"localhost:{port}"],
capture_output=True,
text=True,
)
assert deploy.returncode == 0, f"deploy-ha.sh failed: {deploy.stdout}\n{deploy.stderr}"
log_file = deploy_dir / "logs" / "ctrl-1.log"
proc = subprocess.Popen(
[
str(controller_bin),
"--myname", "ctrl-1",
"--peers", f"ctrl-1=localhost:{port}",
"--workdir", str(deploy_dir / "controller"),
"--certdir", str(deploy_dir / "certs"),
"--logfile", str(log_file),
"--vcpus", "4",
],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
try:
_wait_for_port("localhost", port, _READY_TIMEOUT_S)
yield f"localhost:{port}", log_file
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=10)
shutil.rmtree(deploy_dir, ignore_errors=True)
def test_live_config_push_against_real_single_node_controller(live_controller, tmp_path: Path) -> None:
"""Real ``xceadmin config build`` + real ``xceadmin config push`` against
a real deployed ``xce_controller`` over actual TLS+capnp -- no mock
anywhere (Requirements 2.1, 2.5, 3.1).
Asserts (module docstring has the full rationale):
- no crash decoding the real reply (task 6.1's regression lock -- the
actual point of this test)
- a parsed ``versionTag``/``ackedCount`` are present in the output
(2.5)
- the REAL observed single-node exit/ackedCount semantics (ackedCount
== 0, exit 1, "push rejected") rather than a hypothetical exit 0
- the controller's own log independently confirms the config
genuinely committed, upgraded, and activated server-side (2.6),
decoupled from the CLI's single-node-blind ackedCount heuristic
"""
controller_addr, log_file = live_controller
xceadmin_bin = _find_first_executable(_XCEADMIN_CANDIDATES)
if xceadmin_bin is None:
pytest.skip("no xceadmin binary found under xce.build/build.release/build.dbg")
cluster_bin = tmp_path / "cluster.bin"
build_result = subprocess.run(
[str(xceadmin_bin), "config", "build", str(REPO_CONFIGS_DIR), "-o", str(cluster_bin)],
capture_output=True,
text=True,
)
assert build_result.returncode == 0, build_result.stderr
assert cluster_bin.is_file()
push_result = subprocess.run(
[str(xceadmin_bin), "config", "push", str(cluster_bin), "--controller", controller_addr],
capture_output=True,
text=True,
)
# Task 6.1's regression lock: the real controller reply must decode
# cleanly through the Request{body.replymsg} envelope unwrap -- no
# capnp KjException, no unhandled traceback, against REAL wire bytes
# (not a mock fixture that could silently drift from the real shape).
assert "KjException" not in push_result.stderr, push_result.stderr
assert "Traceback" not in push_result.stderr, push_result.stderr
match = _VERSION_TAG_RE.search(push_result.stdout)
assert match, f"no versionTag/ackedCount found in stdout: {push_result.stdout!r}"
version_tag, acked_count = int(match.group(1)), int(match.group(2))
assert version_tag > 0
# REAL, live-verified single-node behavior (module docstring): a
# genuinely peer-less controller has no standbys to ACK, so
# acked_count is always 0 and xceadmin.cli's `acked_count > 0`
# postcondition reports this as a rejection -- even though, per the
# controller-log assertions below, the push itself fully succeeded.
assert acked_count == 0
assert push_result.returncode == 1
assert "push rejected" in push_result.stderr
log_text = log_file.read_text()
version_tag_hex = f"{version_tag:016x}"
assert f"tag={version_tag_hex}" in log_text, log_text
assert "Config change committed" in log_text
assert f"Config upgraded to tag={version_tag_hex}" in log_text
assert f"config activated from committed marker tag={version_tag_hex}" in log_text