test(ha-commit-barrier-rollout): fact-retention-across-quorum-loss proof (task 4.3)

Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
This commit is contained in:
Rongsong Shen 2026-06-12 15:29:47 +00:00
parent 4fafeadbbc
commit 732e836868
6 changed files with 594 additions and 312 deletions

View file

@ -60,7 +60,7 @@
- Observable completion: all task 4.1 tests PASS; existing gate tests for request opcodes unchanged
- _Requirements: 3.2_
- [ ] 4.3 Write the fact-retention-across-quorum-loss integration test
- [x] 4.3 Write the fact-retention-across-quorum-loss integration test (bug-396 end-to-end evidence complete)
- Scenario (stub registry, n_peers=3): disconnect standbys (quorum lost) → drive worker-fact paths (job start/finish via `do_notify`-level calls) → appends succeed with no error, entries pending (`lsn > committed_lsn`), nothing dropped; reconnect + supply covering ACKs → `committed_lsn` advances over the retained suffix with no re-delivery of the facts; assert no timeout was ever applied to the fact entries
- Observable completion: test PASSES against tasks 1.2 + 4.2 output (if it fails, fix the RETAIN/gate path, not the test); retained-depth observability counter visible in test logs
- _Requirements: 3.1, 3.3, 3.4_

View file

@ -1,7 +1,7 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T11:16:47.642Z
> Files: 513 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T15:29:40.114Z
> Files: 515 tracked | Anatomy hits: 0 | Misses: 0
## ./
@ -253,7 +253,7 @@
- `requirements.md` — Requirements Document (~3181 tok)
- `research.md` — Research & Design Decisions: ha-commit-barrier-rollout (~3672 tok)
- `spec.json` (~143 tok)
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~4653 tok)
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~4663 tok)
## .kiro/specs/ha-commit-engine/
@ -706,6 +706,7 @@
## src/controller/
- `ha.c` — define _GNU_SOURCE (~43480 tok)
- `job.c` — include <errno.h> (~8196 tok)
- `join.c` — include "config.h" (~2530 tok)
- `meson.build` (~1134 tok)
@ -715,4 +716,5 @@
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1368 tok)
- `test_commit_barrier_reply.c` — Task 3.1 commit-first-then-reply integration tests + task 3.3 receipt-ACK verdict pins (jobdispatch/preason/rusage); cluster harness w/ real persist, wired commit engine, stub standbys, reply-capture endpoint, notify/get_ep_worker capture stubs (~17500 tok)
- `test_commit_barrier_reply.c` — Commit-barrier reply-ordering integration tests (tasks 3.1/3.3/4.3): four restructured handlers, receipt-ACK verdict pins, fact-retention-across-quorum-loss (~21000 tok)
- `test_commit_barrier_rollout.c` — Declares explicitly (~11136 tok)

File diff suppressed because it is too large Load diff

View file

@ -9,6 +9,8 @@
<!-- How the user likes things done. Code style, tools, patterns, communication. -->
## Key Learnings
- Census EXACT method (2026-06-12): count DISTINCT Unity `file:line:test:PASS` triples (`grep -aoE` + sort -u), not raw `2mPASS` occurrences — tinylog interleaving garbles ~9 verdict lines per run and duplicates others; distinct-triple count matched the 494-baseline arithmetic exactly (495 after task 4.3's +1).
- repl registry reconnect semantics: repl_remove_standby vacates the slot (ep=NULL, keeps last_disconnected_lsn guard); repl_add_standby reuses the first empty slot and memsets it (acked_lsn=0). With both standbys removed, majority_candidate_locked zero-fills to n_peers-1 → candidate 0 → committed_lsn frozen (RETAIN entries safely pending); after re-add, ONE covering ack = 2-of-3 majority.
- repl_wait_majority_ack walAck track is engine-only since barrier-rollout 2.2: entry_lsn==0 returns 0 immediately (documented once-per-lifetime LSN-0 first entry), everything else delegates to commit_submit unconditionally — ce==NULL surfaces as -EINVAL from commit_submit (unreachable in production; cluster_init_phase2 always wires the engine). The counting loop that remains in the function is configAck-ONLY (no self-count, n_active==0 shortcut is Req 8.5 config behavior). persist_wal_sync_append's residual non-engine branch can only store committed_lsn for the LSN-0 no-op case. (2026-06-12, task 2.2)
- `commit_submit(COMMIT_MODE_RETAIN)` is fully lock-free: it returns 0 BEFORE `pthread_mutex_lock(&ce->lock)` (commit.c:233-239), so it is safe to call from hook functions even while entity write locks are held — only WAIT mode takes ce->lock. (barrier-rollout task 1.2, 2026-06-12)
- ha-commit-engine: once repl.c references commit_* symbols (task 5.2), commit.c MUST be in controller_src (meson.build) or the production xce_controller link fails — the "linkage lands with 5.3" note in tasks.md only held while repl.c had no engine references. Lock order ce->lock → r->lock: repl.c calls engine hooks only AFTER releasing r->lock.

View file

@ -957,3 +957,8 @@
| 07:14 | Edited src/controller/ha.c | expanded (+10 lines) | ~205 |
| 14:35 | task 4.2 validated: gate switch admits notify/update/setpreason/clrpreason; build OK; 21/21 rollout, 494 census ZERO FAIL | src/controller/ha.c | success | ~3k |
| 07:50 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~12 |
| 08:02 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 5→6 lines | ~43 |
| 08:03 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 3 condition(s) | ~3291 |
| 08:03 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→5 lines | ~72 |
| 15:15 | Task 4.3 ha-commit-barrier-rollout: fact-retention-across-quorum-loss integration test (GREEN on landing) — quorum lost via repl_remove_standby x2, 4 worker facts via on_job_start/on_job_finish (cluster.c/do_notify not in test link), RETAIN pending + on-disk exactly-once scan, reconnect + covering ACK advances committed_lsn, no re-delivery, retained-depth=4 printed | src/controller/testsuite/test_commit_barrier_reply.c | census 495 distinct Unity PASS (494+1), ZERO FAIL | ~55k |
| 08:29 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~30 |

View file

@ -58,6 +58,7 @@
#include <rtti.h>
#include <errno.h>
#include <fcntl.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
@ -1244,6 +1245,259 @@ static void test_update_op_rusage_receipt_reply_facts_retained(void)
printf("test_update_op_rusage_receipt_reply_facts_retained: PASS\n");
}
/* ════════════════════════════════════════════════════════════════════════
* Fact retention across quorum loss (task 4.3, Req 3.1, 3.3, 3.4)
*
* GREEN-on-landing integration proof of the RETAIN persistence semantics
* (umbrella Req 2.8 end-to-end evidence; completes the bug-396 fix):
* with quorum LOST (both standby registry slots disconnected via
* repl_remove_standby) the worker-fact paths still append, with no error
* and no commit timeout; the entries stay pending (lsn > committed_lsn)
* and nothing is dropped (each fact on disk exactly once); after
* reconnect + a covering ACK, committed_lsn advances over the retained
* suffix with NO re-delivery of the facts (the log itself is the queue).
*
* It lives HERE (not test_commit_barrier_rollout.c) because it needs this
* file's full harness: cluster_t with jobmgr + real tmpdir persist with
* registered production hooks + wired engine (n_peers=3) + the stub
* standby registry + the job.c link (cbrp_make_job).
*
* Facts are driven via on_job_start / on_job_finish (job.c) DIRECTLY:
* do_notify lives in cluster.c, which is NOT in the unit-test link
* (meson controller_unit_test_src), and on_job_* is exactly do_notify's
* fact path minus the opcode decode. Gate admission of OPCode_notify
* during quorum loss is task 4.1/4.2's coverage (test_commit_barrier_
* rollout.c fact-gate cases); this test's focus is RETAIN persistence.
*
* mark_job_* is currently UNGUARDED (idempotency guards land in task 6),
* so only distinct forward transitions are driven: job 80 start→finish,
* job 81 start→finish — four distinct (job, transition) facts, no
* duplicates, independent of future guard behavior.
* ════════════════════════════════════════════════════════════════════════ */
/* Count WAL entries with (wal_type, wal_jobstate_t.jobid) by direct scan
* of the active on-disk segment (persist_wal_async pwrites synchronously,
* so same-process reads see every append without an fsync). The priming
* entry (payload "p", 1 byte) and any non-jobstate entries are skipped by
* the payload-size check. Single segment is guaranteed here: rotation
* needs 64 MB (WAL_SEGMENT_SIZE_BYTES). Returns the match count;
* *state_out (optional) receives the LAST matching entry's state. */
static int cbrp_count_jobstate_entries(persist_t *p, uint32_t wal_type,
uint64_t jobid, uint32_t *state_out)
{
char seg_path[sizeof(p->seg_path)];
uint64_t seg_end, off = 0;
int fd, count = 0;
pthread_mutex_lock(&p->lock);
memcpy(seg_path, p->seg_path, sizeof(seg_path));
seg_end = p->seg_offset;
pthread_mutex_unlock(&p->lock);
fd = open(seg_path, O_RDONLY);
TEST_ASSERT_TRUE_MESSAGE(fd >= 0, "active WAL segment must be openable");
while (off + sizeof(wal_entry_hdr_t) <= seg_end) {
wal_entry_hdr_t hdr;
TEST_ASSERT_EQUAL_INT_MESSAGE(
(int)sizeof(hdr),
(int)pread(fd, &hdr, sizeof(hdr), (off_t)off),
"WAL segment scan: short header read");
if (hdr.type == wal_type
&& hdr.payload_len == sizeof(wal_jobstate_t)) {
wal_jobstate_t js;
TEST_ASSERT_EQUAL_INT_MESSAGE(
(int)sizeof(js),
(int)pread(fd, &js, sizeof(js), (off_t)(off + sizeof(hdr))),
"WAL segment scan: short payload read");
if (js.jobid == jobid) {
count++;
if (state_out != NULL)
*state_out = js.state;
}
}
off += sizeof(hdr) + hdr.payload_len;
}
close(fd);
return count;
}
/* Drive one fact call: it must succeed (rc 0 — in particular NEVER
* -ETIMEDOUT) and return latency-bounded. sync_timeout_ms is 5000 in this
* test, so a fact path that wrongly ran a WAIT barrier with ACKs
* unobtainable would block ~5000 ms and trip the 1000 ms bound — the
* "no timeout was ever applied" assertion (Req 3.1: facts have NO commit
* timeout; RETAIN returns immediately). */
#define CBRP_DRIVE_FACT(call, what) \
do { \
uint64_t t0_ = cbrp_mono_ms(); \
int rc_ = (call); \
uint64_t el_ = cbrp_mono_ms() - t0_; \
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc_, \
what ": fact append must succeed with no error during quorum " \
"loss — never -ETIMEDOUT (Req 3.1, 3.2)"); \
TEST_ASSERT_TRUE_MESSAGE(el_ < 1000, \
what ": no commit timeout may ever be applied to a fact entry " \
"— RETAIN returns immediately (Req 3.1)"); \
} while (0)
/* makeJob deep-copies submit; statics survive Unity's longjmp. */
static job_submit_data_t g_fr_submit80;
static job_submit_data_t g_fr_submit81;
static void test_fact_retention_across_quorum_loss(void)
{
static unsigned int fr_pids[2] = { 8001, 8101 };
job_pidinfo_t start80, start81;
xce_jobfinish_t fin80, fin81;
uint64_t lsn_before, lsn_after, first_fact_lsn, watermark;
uint64_t committed, deadline;
uint32_t st_start80 = 0, st_fin80 = 0;
uint32_t st_start81 = 0, st_fin81 = 0;
/* Arrange: 3-member cluster, two running jobs-to-be. The large
* sync_timeout (5000 ms) gives the CBRP_DRIVE_FACT latency bound its
* discriminating power (see macro comment). */
cbrp_setup(5000);
memset(&g_fr_submit80, 0, sizeof(g_fr_submit80));
memset(&g_fr_submit81, 0, sizeof(g_fr_submit81));
cbrp_make_job(80, &g_fr_submit80);
cbrp_make_job(81, &g_fr_submit81);
/* Act 1 — QUORUM LOST: disconnect both standby registry slots
* (repl_remove_standby = the RDHUP disconnect path's registry effect).
* n_peers stays 3 in the engine: self alone is 1 of 3, no majority. */
repl_remove_standby(&g_hc.persist.repl, &g_hep_a);
repl_remove_standby(&g_hc.persist.repl, &g_hep_b);
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
first_fact_lsn = lsn_before; /* persist_wal_async assigns current_lsn */
/* Act 2 — drive four distinct worker-fact transitions with zero
* quorum: job 80 start/finish, job 81 start/finish (do_notify-level
* fact paths; see section header for why on_job_* directly). */
memset(&start80, 0, sizeof(start80));
start80.jobid = 80;
start80.npids = 1;
start80.pids = &fr_pids[0];
CBRP_DRIVE_FACT(on_job_start(&g_hc, &start80), "on_job_start(80)");
memset(&start81, 0, sizeof(start81));
start81.jobid = 81;
start81.npids = 1;
start81.pids = &fr_pids[1];
CBRP_DRIVE_FACT(on_job_start(&g_hc, &start81), "on_job_start(81)");
memset(&fin80, 0, sizeof(fin80));
fin80.jobid = 80;
fin80.exit_code = 0; /* → JOB_STAT_DONE */
CBRP_DRIVE_FACT(on_job_finish(&g_hc, &fin80), "on_job_finish(80)");
memset(&fin81, 0, sizeof(fin81));
fin81.jobid = 81;
fin81.exit_code = 0;
CBRP_DRIVE_FACT(on_job_finish(&g_hc, &fin81), "on_job_finish(81)");
/* Assert — pending, nothing dropped (Req 3.1, 3.3): all four facts
* appended exactly once, none committed (committed_lsn untouched by
* the membership shrink: zero countable standbys → candidate 0). */
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
watermark = lsn_after - 1;
committed = atomic_load(&g_hc.persist.committed_lsn);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before + 4, lsn_after,
"each of the four facts must append exactly one WAL entry "
"(accepted, not rejected, during quorum loss — Req 3.2)");
TEST_ASSERT_TRUE_MESSAGE(committed < first_fact_lsn,
"every fact entry must remain pending (lsn > committed_lsn) "
"while quorum is lost (Req 3.1)");
/* Retained-depth observability counter (design §Observability:
* committed-vs-appended lag), visible in the test log. */
printf("test_fact_retention_across_quorum_loss: retained-depth=%lu "
"(appended watermark=%lu, committed_lsn=%lu)\n",
(unsigned long)(watermark - committed),
(unsigned long)watermark, (unsigned long)committed);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(4, watermark - committed,
"retained depth must equal the four uncommitted fact entries");
/* Nothing dropped: each fact is on disk exactly once, with the
* post-transition state recorded (start → RUN, finish(0) → DONE). */
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 80,
&st_start80),
"job 80 START fact must be retained exactly once (Req 3.3)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 81,
&st_start81),
"job 81 START fact must be retained exactly once (Req 3.3)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 80,
&st_fin80),
"job 80 FINISH fact must be retained exactly once (Req 3.3)");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 81,
&st_fin81),
"job 81 FINISH fact must be retained exactly once (Req 3.3)");
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_start80);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, st_start81);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_fin80);
TEST_ASSERT_EQUAL_UINT32(JOB_STAT_DONE, st_fin81);
/* Act 3 — RECONNECT + covering ACK (Req 3.4): re-register both
* standbys (slots were vacated; repl_add_standby resets their
* watermarks to 0) and deliver one ACK covering the retained suffix —
* self + sb_a = 2 of n_peers=3, a majority. */
TEST_ASSERT_TRUE(
repl_add_standby(&g_hc.persist.repl, &g_hep_a, "sb_a", 7400) >= 0);
TEST_ASSERT_TRUE(
repl_add_standby(&g_hc.persist.repl, &g_hep_b, "sb_b", 7401) >= 0);
repl_on_ack(&g_hc.persist.repl, &g_hep_a, watermark);
deadline = cbrp_mono_ms() + 2000;
while (atomic_load(&g_hc.persist.committed_lsn) < watermark
&& cbrp_mono_ms() < deadline)
cbrp_sleep_ms(5);
/* Assert — committed without re-delivery (Req 3.4): committed_lsn
* covers the retained suffix; the WAL still holds each fact exactly
* once (no duplicate appends) and current_lsn never moved — the
* RETAINED entries committed, not re-delivered copies. */
TEST_ASSERT_TRUE_MESSAGE(
atomic_load(&g_hc.persist.committed_lsn) >= watermark,
"a covering ACK after reconnect must advance committed_lsn over "
"the retained fact suffix (Req 3.4)");
pthread_mutex_lock(&g_hc.persist.lock);
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_after, g_hc.persist.current_lsn,
"commitment must consume the retained entries — no re-delivery, "
"no duplicate appends (Req 3.4)");
pthread_mutex_unlock(&g_hc.persist.lock);
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 80, NULL),
"job 80 START must still be on disk exactly once after commit");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_START, 81, NULL),
"job 81 START must still be on disk exactly once after commit");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 80, NULL),
"job 80 FINISH must still be on disk exactly once after commit");
TEST_ASSERT_EQUAL_INT_MESSAGE(1,
cbrp_count_jobstate_entries(&g_hc.persist, WAL_JOB_FINISH, 81, NULL),
"job 81 FINISH must still be on disk exactly once after commit");
cbrp_teardown();
printf("test_fact_retention_across_quorum_loss: PASS\n");
}
/* ── Test group entry point ──────────────────────────────────────────── */
void test_commit_barrier_reply(void)
@ -1277,5 +1531,8 @@ void test_commit_barrier_reply(void)
RUN_TEST(test_set_preason_receipt_reply_fact_retained);
RUN_TEST(test_clr_preason_receipt_reply_fact_retained);
RUN_TEST(test_update_op_rusage_receipt_reply_facts_retained);
/* Fact retention across quorum loss (task 4.3, Req 3.1, 3.3, 3.4) —
* green on landing against tasks 1.2 + 4.2 output */
RUN_TEST(test_fact_retention_across_quorum_loss);
UNITY_END();
}