lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(raft-apply-decoupling): kill-mid-fact-stream soak + close bug-1416 boot-time TOCTOU (task 7.2)

Browse source 
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.

Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.

Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
This commit is contained in:
Rongsong Shen 2026-07-07 03:03:40 +00:00
parent 5f98d1b0ef
commit a5e29a6ffa
17 changed files with 74793 additions and 156 deletions
Download patch file Download diff file Expand all files Collapse all files

56
src/controller/testsuite/waldump.c
View file

@ -84,6 +84,43 @@ static int decode_submit_info(const uint8_t* payload, uint32_t len,
return 0;
}
/* True for the WAL types whose payload is the flat, non-capnp
* wal_jobstate_t {jobid, state, _pad} (wal_types.h) — every worker-reported
* job-state fact (START/FINISH/FAIL/STOP/RESUME/UPDATE/RUSAGE/CKPT/PREASON).
* Added for raft-apply-decoupling task 7.2: the kill-mid-fact-stream live
* regression needs to verify, per jobid, which fact WAL entries actually
* committed on each node — before this, waldump only decoded jobid for
* WAL_JOB_SUBMIT, leaving every fact entry printed as "jobid=?". */
static int wal_type_is_jobstate(uint32_t type)
{
switch (type) {
case WAL_JOB_START:
case WAL_JOB_FINISH:
case WAL_JOB_FAIL:
case WAL_JOB_STOP:
case WAL_JOB_RESUME:
case WAL_JOB_UPDATE:
case WAL_JOB_RUSAGE:
case WAL_JOB_CKPT:
case WAL_JOB_PREASON:
return 1;
default:
return 0;
}
}
/* Decode a wal_jobstate_t payload. Native-endian flat struct (single-node WAL
* — see wal_types.h), so this is a bounds-checked copy, not a parser. Returns
* 0 on success, -1 if the payload is too short to hold the struct. */
static int decode_jobstate_info(const uint8_t* payload, uint32_t len,
wal_jobstate_t* out)
{
if (!payload || len < sizeof(*out))
return -1;
memcpy(out, payload, sizeof(*out));
return 0;
}
/* True if `dir` directly contains at least one seg-*.wal file. */
static int dir_has_segments(const char* dir)
{
@ -342,6 +379,25 @@ static void scan_segment(const char* seg_path, FILE* out,
}
*prev_submit_jobid = jobid;
*prev_submit_pcrc = pcrc;
} else if (wal_type_is_jobstate(hdr.type)) {
/* Worker-reported job-state fact (task 7.2): decode the flat
* {jobid, state, _pad} payload so a per-jobid committed-fact check
* (e.g. "did WAL_JOB_FINISH for jobid X actually commit on this
* node") can grep waldump output directly, the same way the
* existing WAL_JOB_SUBMIT branch above already supports
* expect-job/joblist cross-checks. */
wal_jobstate_t js;
if (decode_jobstate_info(payload, hdr.payload_len, &js) == 0) {
char jidStr[128];
if (fmt_jobid(jidStr, sizeof(jidStr), js.jobid) < 0)
snprintf(jidStr, sizeof(jidStr), "%" PRIu64, js.jobid);
snprintf(info, sizeof(info), "jobid=%s state=0x%x pad=0x%x",
jidStr, js.state, js._pad);
} else {
snprintf(info, sizeof(info), "jobid=?");
}
/* a non-submit entry breaks the adjacency for dup detection */
*prev_submit_jobid = 0;
} else {
/* a non-submit entry breaks the adjacency for dup detection */
*prev_submit_jobid = 0;