wdgzyjx/myxce
1
0
Fork 0
forked from lei.chen/myxce
Code Pull requests Activity

test(ha-commit-barrier-rollout): RED commit-first-then-reply tests per opcode (task 3.1)

Browse source 
New test_commit_barrier_reply.c drives do_jobcontrol_op /
do_queuecontrol_op / do_hostcontrol_op / handle_worker_join directly
(pthread harness, ep->outq reply capture, wired engine n_peers=3).
12 cases: 8 RED (reply-before-commit + bug-447 pins), 4 validation
pins green. DISCOVERY bug-447 (reviewer-confirmed): key_find found-rc
leak means queue/host control never append WAL in production — 3.2's
rc normalization is the fix point. Census 484 = 472 + 12; zero
baseline regressions.
This commit is contained in:
Rongsong Shen 2026-06-12 10:00:38 +00:00
parent 8f688eace1
commit e1c3c173f4
8 changed files with 1036 additions and 140 deletions
Download patch file Download diff file Expand all files Collapse all files

4
.kiro/specs/ha-commit-barrier-rollout/tasks.md
View file

@ -32,7 +32,7 @@
- _Requirements: 1.1, 1.2, 1.3_
- [ ] 3. Reply-ordering restructure of request-shaped handlers
- [ ] 3.1 Write RED integration tests for commit-first-then-reply per opcode
- [x] 3.1 Write RED integration tests for commit-first-then-reply per opcode
- Tests drive the four handlers directly (vtask-context test harness, stub standby registry): for each of `do_jobcontrol_op` (not-started-job branch, producing WAL_JOB_UPDATE/WAL_JOB_FAIL), `do_queuecontrol_op`, `do_hostcontrol_op`, `handle_worker_join` — with ACKs withheld the requester reply carries an error (`-ETIMEDOUT`) and no success reply is ever observable while the operation's entries exceed `committed_lsn`; with ACKs supplied the success reply is observable only after `committed_lsn` covers the entries; local validation failures still reply immediately with their existing error codes
- Observable completion: new cases FAIL (red) against the current reply-before-commit handlers (anchors: job.c:932, queue.c:130, worker.c:700, join.c:107-109 — re-verify with grep)
- _Requirements: 1.1, 1.2, 1.3, 1.4_
@ -112,5 +112,7 @@
- (1.1) wal_entry_type has 13 values (wal_types.h:44-58), no WAL_NOOP yet (lands in ha-single-leader-role, which must add a classification row + sweep entry — the totality sweep will catch it).
- (1.1) Suite baseline is ZERO failures since ha-commit-engine 6.1. detect_leaks=0 remains the harness norm (6 inventoried pre-existing leak groups).
- (1.2) CENSUS METHOD: only ~9 of 48 groups emit Unity summary blocks; the rest report per-test PASS lines only — count distinct Unity results (~465) and grep FAIL lines, never sum summary blocks. commit_submit(RETAIN) early-returns BEFORE taking ce->lock (commit.c:233-239) — registration is lock-free, safe with entity write locks held. wal_hook_append_classified is the single classified-append seam; appended_lsn read = current_lsn - 1 under p->lock (benign race, commented).
- (3.1) **bug-447 CONFIRMED (reviewer-verified to librtti disassembly)**: key_find returns 1 on found (unnormalized from xkvmgr_find/kvtab_find); queue_ctrl (queue.c:61, gate :106) and worker_ctrl (worker.c:644, gate :678) leak rc=1 → AFTER_EVENT never fires on the found path → **queue/host control silently unpersisted+unreplicated in production today**; job_ctrl appends via mark_job_* hooks but still replies status 1. Fix point = 3.2's handler rc normalization (key_find is in binary librtti, out of repo). RED tests pin it.
- (3.1) Test link gained job.c/queue.c/join.c (test-only meson change) + 8 inert link stubs in test_commit_barrier_reply.c (none on tested paths — reviewer-traced). Reply capture = zeroed stack endpoint ep->outq inspection (test_config_repl.c:2210 precedent; endpoint_update_epoll no-ops when iomgr NULL). RED-window orphaned fsync threads self-resolve at GREEN (reviewer finding 5).
- (2.2) Legacy walAck counting RETIRED: walAck track = entry_lsn==0 → 0 (LSN-0 interim) else unconditional commit_submit (ce==NULL → -EINVAL, fail-loud). configAck track byte-equivalent (task 5's surface intact, incl. its n_active==0 shortcut and acked_config_lsn>0 counting — both die at task 5). persist_commit_barrier: ce==NULL → 0 (vacuous, documented). test_repl_unit's two lsn=0 walAck callers are now vacuous for quorum counting (INFO for task 5/7.2). CENSUS = 472 (465 at 1.2 + 7 from 2.1).
- (2.1) BARRIER WATERMARK RESOLVED (reviewer-judged): design's literal "Snapshot lsn = p->current_lsn" is loose phrasing contradicted by its own anchors — the coherent contract is watermark = current_lsn - 1 (last appended; persist.h:71 invariant committed_lsn <= current_lsn - 1 makes the literal reading unsatisfiable). Tests pin current_lsn - 1 with off-by-one killing power in BOTH directions; task 2.2 must implement that. Resolved contract recorded in persist.h doc comment.

55
.wolf/anatomy.md
View file

@ -1,7 +1,7 @@
# anatomy.md
> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T04:13:06.312Z
> Files: 511 tracked | Anatomy hits: 0 | Misses: 0
> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T10:00:34.785Z
> Files: 509 tracked | Anatomy hits: 0 | Misses: 0
## ./
@ -187,7 +187,7 @@
- `.gitignore` — Git ignore rules (~47 tok)
- `codegraph.db-shm` (~8738 tok)
- `daemon.log` (~3298 tok)
- `daemon.log` (~3580 tok)
- `daemon.pid` (~44 tok)
## .kiro/settings/rules/
@ -253,7 +253,7 @@
- `requirements.md` — Requirements Document (~3181 tok)
- `research.md` — Research & Design Decisions: ha-commit-barrier-rollout (~3672 tok)
- `spec.json` (~143 tok)
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~3743 tok)
- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~4337 tok)
## .kiro/specs/ha-commit-engine/
@ -261,7 +261,7 @@
- `design.md` — Design: ha-commit-engine (~7600 tok)
- `requirements.md` — Requirements Document (~2715 tok)
- `research.md` — Research Log: ha-commit-engine (~1345 tok)
- `spec.json` (~140 tok)
- `spec.json` (~255 tok)
- `tasks.md` — Implementation Plan: ha-commit-engine (~4593 tok)
## .kiro/specs/ha-config-bootstrap/
@ -393,14 +393,14 @@
## graphify-out/
- `.graphify_labels.json` (~5614 tok)
- `.graphify_labels.json` (~5762 tok)
- `.graphify_root` (~1 tok)
- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-12) (~45713 tok)
- `manifest.json` (~35154 tok)
- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-12) (~46969 tok)
- `manifest.json` (~35204 tok)
## graphify-out/cache/
- `stat-index.json` (~44628 tok)
- `stat-index.json` (~44692 tok)
## graphify-out/cache/ast/
@ -428,6 +428,7 @@
- `053929f384ca0a4fdabda00415d3f0e0a7f8ccbd86f273deadd54ac0590492a2.json` (~18832 tok)
- `0550909cf1a1da30dee6c2653f79809399374ee35919a467be9583fa7d27f5c5.json` (~71978 tok)
- `057d67f05183f6fe7fd3414cb6f46de234ebe58fa3dbca7dddc5441ee3f8a2f1.json` (~6218 tok)
- `061bfdbdc03ae56a916d366f3200ca2c610d9ad141e11682e47243197d93632b.json` (~592 tok)
- `06578221cb7a75ee06e0460d5a42c48037e3ddcebbc21fae45825886ceaa0355.json` (~15340 tok)
- `06a0b286e4c22fb4c981239820aea2042950d1d21dc96de40226b14cc696065e.json` (~8244 tok)
- `07014154947b85f86499894c59be83a316bf3fc006c8fc55c316cb893462977e.json` (~3083 tok)
@ -478,6 +479,7 @@
- `151a099eec558a9e4c89bc08aae88ba096681971be1a94c7625343fd9a3a88e5.json` (~7640 tok)
- `15224225923ce6697e098312e1dc8429b2768b415534886f344c212446813d83.json` (~495 tok)
- `152d9c2445a8247d38b52f6ecab0cf6f36ba144aeb93e0bc7603445e7ed2c6cc.json` (~5075 tok)
- `155212874dafa331b4fd417f7ebd9bafae70754b134e49c5cfb694a47c21def5.json` (~81 tok)
- `157041af18754d4729c7ead7ed02190c14fa052b3c65deac29309014894e67a8.json` (~10023 tok)
- `157c7929f5a2274e2f98c623d21343ab2f01711d4fd4e742caef92e2bf2b3463.json` (~4857 tok)
- `15cc2d188fbb7c7bc937a39311efdc8cd2536b01947309e45f5563c0e40a7923.json` (~1376 tok)
@ -505,10 +507,13 @@
- `1c86ed59f0918f5ea0f47f54917854eb2ce2a9b9b6724a107c9237ed0a7f7ea2.json` (~2916 tok)
- `1c906780916fd91e042adaa801f385dc55bc1f29c851908ad61f8c5db0040107.json` (~2163 tok)
- `1c9619334c216e67eec7d7a1c420d26476001173e926088e46d4d2acdd52f8d4.json` (~320 tok)
- `1d7c4a75171c44fea2a337b2b8756a0b4aefd3407e5a0cce07089b2722f4939d.json` (~7794 tok)
- `1dcf7b97c92f5149c264d18f36fb408c96372770c52491ec5d320b4a17533612.json` (~8110 tok)
- `1e8313e77c1ec1f11b656b60fdcefe81f0d61322c4cdcbf51cf918c7e7965420.json` (~15340 tok)
- `1ef77ec32c809ea9f6abe455804f5630dc3a9693e633b03068f3546dfeeab099.json` (~3149 tok)
- `1f6c6535ebbdcedd95bf44e6b96f07f1234d7ea54cb747d6e113f90ccf4bb7ea.json` (~13910 tok)
- `1f7e683af7e73fefdcf14ce40c2ee81dfa8aa113b7aa90b51067d870710baa7a.json` (~332 tok)
- `1f9c1094d37b3b77b52d3863f257184175c1f040ed46cfbd75b54598f428d466.json` (~81 tok)
- `1fdd469bb6d51958ab2ba15266b745abf7014618c2055cf5b098276ea4a501bd.json` (~62630 tok)
- `1fe907805f188b6cf4e490bbab545a31e875b83a02bb0e7220e1b8666ddee3e6.json` (~332 tok)
- `20456a4402821d8eeb5ccfb2b27a16965b9d4ed85058de966a1f4894fe8443eb.json` (~9852 tok)
@ -526,6 +531,7 @@
- `23ae45376874ad36b52972b9b9031b66aa76f34d1de4b793801908d2dc0bdbc4.json` (~4128 tok)
- `23b8085ed43aaaed2d876c21ecb494fa28af40a1cefaffcf5ffc4b131d9ed3f8.json` (~1662 tok)
- `23cbeb01ae9cd9669fae81ddb2b2d6c7f89abd17b12365a5f6b256a32a717616.json` (~490 tok)
- `24272ebb7e242bd439e42b571de9c4b67ba0f5a3a09c07bde630b6050cbeef4a.json` (~15340 tok)
- `242faeda0abe77b1e66a515917a267528fe2d1b0c091dbced56e493e9cf69737.json` (~3934 tok)
- `24fa3f5b3b5b6d59088387aad4130a5e2d657f88706205d6dbe57e8afda2e425.json` (~722 tok)
- `251ce70fbb55f062faef6259731b4c82238f4f7af22d2c1ed266c86c15263b99.json` (~502 tok)
@ -545,6 +551,7 @@
- `2b0b0f1eb49df36a2e8fd4f88317124dd1edf9a0f08545a1a53accd6b47b16b1.json` (~309 tok)
- `2b9e4e051f5e39e6df4dc1c1ff7611d9a3590262f41b88e978fa664c596524b8.json` (~269 tok)
- `2baff6d343f2ec98cec20237f085bb8804623059aa145279f05e1d921aa1f8bc.json` (~68072 tok)
- `2bc3ed1bf9f42caac3f6b63f8e317f329f2705320f02a330b243e1b605c6bc9b.json` (~76991 tok)
- `2cbca2b1bf0345399a27e19131777f4d537a479e5d9eea2aefe60bfa6f0c848d.json` (~5279 tok)
- `2d59df43c2cc4d09ea8566c4a94d70cf1a59b8d86e8fc4f9d7c49e92fd356d75.json` (~702 tok)
- `2e097eddc32d49647dfd7b1c33bb454265b5587ec5bf148029c1ea496321073c.json` (~5588 tok)
@ -574,6 +581,7 @@
- `35f823871a12ed1809e20d70a663c8a8929fb35f3748705499e32df6cc4ec5c6.json` (~277 tok)
- `35fd2a3ceb004b1d47fe5ccc509be97f4d380faebcfff32399237e9dfab59c00.json` (~6584 tok)
- `367e004bbee4bd3a32e05eaf817f2dae69e39266c36b1649e4d3929362e383da.json` (~930 tok)
- `36fa47b81d63c37de1102693c833e5a1c113dcb414837b3643ba68ff94058545.json` (~592 tok)
- `3702a2af17c73eed0470df46cecddc0b4c9a93aaab893ff5292d5c5b9b4ed512.json` (~9636 tok)
- `3714706b1961d4945195e8613ba7a5fcbf550c040e671d5860f2e8d747dc477c.json` (~6708 tok)
- `37a543436eff088729eaa2a790c9ef9e58f59f226122e12aa2a0e6c38bc6f5a0.json` (~1870 tok)
@ -596,6 +604,7 @@
- `3d0a807870ec7f6703eab25022ba70e6d12cca273ef820ea6e3bc8a31c97afa5.json` (~10333 tok)
- `3d7e3a102cd4fc3bc27f599364f8f63ec75e6cb1ea3c47bcacc0e1261b485a15.json` (~3783 tok)
- `3da9ee357a09d89a26df59bd0393606349c2d0849078091078924f60b5cdfd51.json` (~495 tok)
- `3db60e6e67d5bb6410285138cb4c838b8adb1d211ebe6ee0c4938820fad4c66b.json` (~269 tok)
- `3db8027a5cdc79633520de23a59b83fb76ecfba3132ba8a4f52e785220157e70.json` (~336 tok)
- `3ddf0434ea5d953052cfdb1597b14704a3692537f86a9fc442ce21024160eb1a.json` (~30420 tok)
- `3de5f25efab468345f714829408944025550d1056ba2b46fa3f3f29bcd5a9573.json` (~2258 tok)
@ -607,7 +616,9 @@
- `3f8d83e0da73654fdad2d36d37d926e0fcf87f3fc645834c75938afed280aa20.json` (~81 tok)
- `3f9cade10a0070913a41048bc2462ee7a4680399174c61c30c8d6e8a6e722cee.json` (~4780 tok)
- `40138b45926e46347669dbe289bb2ea7597aedc0eb558bd0d39aacfef5df62e1.json` (~2333 tok)
- `406549c33ed009f33fce2953bac23493fac1a3b8ddbb7f4ededf56d4d4467d29.json` (~10135 tok)
- `40a5802d6063d7dceabcd49010eac965e03377546e69b2839ce06e3bc19fb99b.json` (~3592 tok)
- `40a70514720bf07cfbe92de67954839e91019b1170af380b1570f710e739a49d.json` (~652 tok)
- `40a727896bbebc2f6e00afb878b8c666f7fdefa3f34fea7215de4ed042087b96.json` (~81 tok)
- `4109287f426902eeb3615a1912430eac8ca65558cc4ec26c781ea237d5460ef0.json` (~28018 tok)
- `41bc353adb8853621ebd82fa735c038f98fe3a7e14d44a2eba6a50f01b4cc944.json` (~1511 tok)
@ -625,6 +636,7 @@
- `45a3f01bceef855fff2282e1e2f5a96edaf6e49eb2dd6774de2c951190939476.json` (~876 tok)
- `461865ad5d4eb698d9f8b0d0189052bd8733835f159fdec76ee474deb74fac89.json` (~466 tok)
- `468a56306bfbb4b7f8f052404e6f4d239bde83c10b206520cb8e9fd6631647c4.json` (~1038 tok)
- `46caff82b95eaec8f55497010d7bd131f4a8651ee883a4d3ea1a121cfe515618.json` (~81 tok)
- `473aad72b667e57806070866e41e43c658d0c08d9422b2bab8641b027e29d2db.json` (~53573 tok)
- `4761c9bb00e54bf17b43a2ebb04ba21b531111c34e93a9dbc957756076e78bf8.json` (~25212 tok)
- `480dbaa39314c68c3d4da3c1cb184846dc98ea3f672516afb1683ad674bc88cf.json` (~678 tok)
@ -658,6 +670,7 @@
- `51103bc49cc87487f9efa30dcffc5f7639f0cc6cbf8acc722f5da33fd0ce32e3.json` (~8308 tok)
- `5175f1392d190fb958eb902f22aea50b3506648b739e0cb8a7bc42d389231e4b.json` (~4502 tok)
- `51d646ce7e4c76a33d29d45d20149ff8825fe4339a842c2d000ed93d9e46f3de.json` (~12490 tok)
- `521cf1ac969a070d624e689274f75a76fc09d84e8a8ba86908b5a6178e776b03.json` (~10023 tok)
- `527377e036844617174eac5b08965176bb127edc5db3d5a77f362b034a4ec4fd.json` (~10319 tok)
- `5286722a3fefdea557784ae42c65304cf227e92c44178fff83b9f2394c57b2e6.json` (~5453 tok)
- `528bfbd499082064cc9b015a4092ffba50f77e09f2f90b5497e20aec9aba24fb.json` (~138 tok)
@ -690,30 +703,12 @@
- `5a822398dceebb814bc41d8ffb801c51f89a129f9d090965877f7f67782a1b23.json` (~4032 tok)
- `5a92249e0d4f9dd34ff89497b4c540e6b84a672a455b0a5be5ca710b94e9eb79.json` (~4454 tok)
- `5a974e6340597c0447c72b0d7bc977a668a8da2fac8c90c810e886c319e9bbd0.json` (~2118 tok)
- `5a99edc88f1f75983b12b9b8931c7ba5717424024e2ffb000e19564f2956175c.json` (~2393 tok)
- `5b5887beb614d310bb85633443a1e29df0dec82a39650daecc39d529ffe5ce37.json` (~4806 tok)
- `5b710141d6eb0f199d33b695898e32519aca05ea857921dd62bf508c7f954948.json` (~567 tok)
- `5b791c1c235d96fe2e512c3d6fedd8f44c5ee256db33beb485935a3f6cc3d1c1.json` (~6930 tok)
- `5c1cd225c770bed4fc03e72d8433eb17a2193588c47d5abeb26580098aab6506.json` (~77 tok)
- `5cc2f13594e5c0d36031a44dc0b88706f3e547d5e48720b55f02a68eb3b3149b.json` (~789 tok)
- `5ce1db6b15faeeb658ef1fca18682532adca25278a2add781eb3c7caf4792b44.json` (~22459 tok)
- `5cebb6a86cbc5a90d803973f1365c1673f3a11bd36cfc5178bdc985a03950378.json` (~8230 tok)
- `5d7b2f244a9e7100f16139db782e80265d4724d5cb53b7c51fab1aeaaf1fb7b6.json` (~1313 tok)
- `5d996b732909e4428bb55522f4fdad491a98075c3aa256d7a816956bd42adac7.json` (~984 tok)
- `5da97b26e48c2ea5c6df990bedce598cfd8052022102fa7a7e4c499c6f1374f5.json` (~1773 tok)
- `5e6d0ac84e99bdd6f6129fa521f73726ded1a47ff2e5d46a7c69026cec7522a5.json` (~591 tok)
- `5eb698f53d93049c137820a52db3d1a7c451be085cea9d956a06b44671034787.json` (~1504 tok)
## src/controller/
- `meson.build` (~1109 tok)
- `persist.c` — define _GNU_SOURCE (~13786 tok)
- `meson.build` (~1130 tok)
## src/controller/testsuite/
- `controller_test.c` — include <argp.h> (~1346 tok)
- `test_commit_barrier_rollout.c` — RED tests for ha-commit-barrier-rollout 1.1: wal_classify totality (verified table), unknown-type fail-fast via fork+SIGABRT, fact-hook RETAIN routing (~4884 tok)
## src/include/
- `persist.h` — pragma once (~2129 tok)
- `controller_test.c` — include <argp.h> (~1368 tok)
- `test_commit_barrier_reply.c` — Task 3.1 RED integration tests: commit-first-then-reply per opcode (jobctrl/queuectrl/hostctrl/worker-join) via endpoint outq reply capture + pthread handler runner; pins the key_find rc==1 hook-skip gap (~8639 tok)

168
.wolf/buglog.json
View file

@ -7046,6 +7046,174 @@
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T00:51:52.673Z"
},
{
"id": "bug-438",
"timestamp": "2026-06-12T03:52:57.142Z",
"error_message": "Incorrect value in code",
"file": "src/controller/meson.build",
"root_cause": "Had 'commit.c'",
"fix": "Changed to 'testsuite/test_commit_barrier_rollout.c'",
"tags": [
"auto-detected",
"wrong-value",
"build"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T03:52:57.142Z"
},
{
"id": "bug-439",
"timestamp": "2026-06-12T04:13:06.326Z",
"error_message": "Missing error handling in unknown",
"file": ".kiro/specs/ha-commit-barrier-rollout/tasks.md",
"root_cause": "Code path had no error handling — exceptions would propagate uncaught",
"fix": "Added try/catch block",
"tags": [
"auto-detected",
"error-handling",
"md"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T04:13:06.326Z"
},
{
"id": "bug-440",
"timestamp": "2026-06-12T04:18:19.158Z",
"error_message": "Missing guard clause",
"file": "src/controller/persist.c",
"root_cause": "No early return/throw for edge case: detail == HOOK_DETAIL_WORKER_JOIN ||\n detail == H",
"fix": "Added guard clause: if (detail == HOOK_DETAIL_WORKER_JOIN ||\n )",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T04:18:19.158Z"
},
{
"id": "bug-441",
"timestamp": "2026-06-12T04:18:31.067Z",
"error_message": "Significant refactor of ",
"file": "src/controller/persist.c",
"root_cause": "4 lines replaced/restructured",
"fix": "Rewrote 7→12 lines (4 removed) | Also: return persist_wal_async(p, wal_type, &js, sizeof(; persist_t* p = cls; | Also: int rc = persist_wal_async(p, wal_type, pkt, (uint; persist_t* p = cls;",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 3,
"last_seen": "2026-06-12T04:19:00.652Z"
},
{
"id": "bug-442",
"timestamp": "2026-06-12T04:50:32.154Z",
"error_message": "Incorrect value in code",
"file": "src/controller/testsuite/test_commit_barrier_rollout.c",
"root_cause": "Had \"xce/types.h\"",
"fix": "Changed to \"xce/errnox.h\"",
"tags": [
"auto-detected",
"wrong-value",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T04:50:32.154Z"
},
{
"id": "bug-443",
"timestamp": "2026-06-12T05:14:36.392Z",
"error_message": "Missing guard clause",
"file": "src/controller/persist.c",
"root_cause": "No early return/throw for edge case: p == NULL",
"fix": "Added guard clause: if (p == NULL)",
"tags": [
"auto-detected",
"guard-clause",
"c"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T05:14:36.392Z"
},
{
"id": "bug-444",
"timestamp": "2026-06-12T09:09:04.495Z",
"error_message": "Significant refactor of ",
"file": "src/controller/testsuite/test_commit_barrier_reply.c",
"root_cause": "2 lines replaced/restructured",
"fix": "Rewrote 3→5 lines (2 removed) | Also: static xce_signal_t g_sigkill = { .signo = SIGKIL; static xce_jobctrl_t g_jobctrl;",
"tags": [
"auto-detected",
"refactor",
"c"
],
"related_bugs": [],
"occurrences": 2,
"last_seen": "2026-06-12T09:09:10.703Z"
},
{
"id": "bug-445",
"timestamp": "2026-06-12T09:30:00Z",
"error_message": "AddressSanitizer SEGV in hash_string via makeResource when running controller_unit_test -t test_commit_barrier_reply standalone",
"file": "src/controller/testsuite/test_commit_barrier_reply.c",
"root_cause": "Global xstring table uninitialised in standalone runs: makeString() returns NULL, resobj name becomes NULL, key_add hashes NULL. In full-suite runs test_worker_join.c:520 initialises the table unguarded, so the bug only appears with -t selection.",
"fix": "Probe makeString(\"cbrp-probe\") in the group entry; call xstring_table_init(139) only when the probe returns NULL (avoids double init in full-suite order).",
"tags": [
"testsuite",
"xstring",
"fixture",
"segv",
"ha-commit-barrier-rollout"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T09:30:00Z"
},
{
"id": "bug-446",
"timestamp": "2026-06-12T09:30:00Z",
"error_message": "makeJob() returns NULL (Expected Non-NULL) in commit-barrier reply fixture",
"file": "src/controller/testsuite/test_commit_barrier_reply.c",
"root_cause": "job_init (lib/jobj.c) returns -EINVAL when xce_job_t.submit == NULL, and makeJobsub requires job_obj_init() module tables which no controller test group initialises.",
"fix": "Fixture calls job_obj_init(64, 64) once per process (static guard) and provides a zeroed static job_submit_data_t as jdef.submit.",
"tags": [
"testsuite",
"jobj",
"fixture",
"ha-commit-barrier-rollout"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T09:30:00Z"
},
{
"id": "bug-447",
"timestamp": "2026-06-12T09:30:00Z",
"error_message": "queue_ctrl/worker_ctrl append NO WAL entry and reply status 1 on success; job_ctrl also returns 1 (key_find found-rc leak)",
"file": "src/controller/queue.c",
"root_cause": "key_find (xkvmgr_find, lib/kvmgr.c:130-151) returns 1 when the key is found. queue_ctrl (queue.c:60), worker_ctrl (worker.c:643) and job_ctrl (job.c:314) store it in rc and never reset it: the 'if (rc == 0)' AFTER_EVENT hook gate is skipped (no WAL_QUEUE_CTRL / WAL_WORKER_CTRL append on the found path) and the requester reply carries status 1 instead of 0. Production gap discovered by task-3.1 RED tests; queue/host control state changes are currently never persisted or replicated.",
"fix": "NOT fixed here (test-only task). Pinned RED by test_commit_barrier_reply.c append assertions; the task-3.2 handler restructure (rc = ctrl fn; if rc==0 barrier; reply rc) must normalise ctrl-fn rc semantics so the hook fires and success replies are 0.",
"tags": [
"production-gap",
"queue_ctrl",
"worker_ctrl",
"job_ctrl",
"key_find",
"wal",
"ha-commit-barrier-rollout",
"red-finding"
],
"related_bugs": [],
"occurrences": 1,
"last_seen": "2026-06-12T09:30:00Z"
}
]
}

3
.wolf/cerebrum.md
View file

@ -41,6 +41,9 @@
- Unity harness quirks in controller_unit_test (2026-06-11): (a) failure lines misattribute the FILE — Unity.TestFile comes from whichever group's UNITY_BEGIN ran, so a failure printed as test_repl_unit.c:294 may live in test_p2_integration.c; locate the actual test by grepping for the test NAME. (b) Each group's UNITY_BEGIN resets counters, so the process exit code only reflects the LAST group — failures in earlier groups are masked (exit=0); meson reports green despite real failures. Grep output for FAIL with `grep FAIL` on the raw log (ANSI codes sit between ':' and 'FAIL', so `grep ":FAIL"` misses them).
- 3 pre-existing controller suite failures on persistent_ha baseline (2026-06-11, verified by stash/rebuild without ha-commit-engine changes, see bug-411): test_committed_lsn_does_not_advance_on_timeout (rc -110 vs expected 0 — persist_wal_sync_append now returns -ETIMEDOUT, test expects old rc==0+sync_rc contract), test_repl_ack_after_promote_gate_allows_jobsub and test_rdhup_primary_dispatch_gate_opens (both -10013: ha_has_write_quorum false, standby_endpoints empty with n_peers=2).
- ha-commit-engine: task 1.2 shipped commit_on_membership_change as a FULL re-evaluation (commit_advance_locked + cond broadcast), and majority_candidate_locked already excludes in_catch_up + ep==NULL slots. Task 3.1 membership tests are green-on-arrival; task 3.2 is a no-op (verified 2026-06-11, tests 17-20 in test_commit_engine.c).
- key_find (xkvmgr_find) returns 1 when found, NOT 0 — queue_ctrl/worker_ctrl/job_ctrl leak this into rc: `if (rc == 0)` hook gates never fire on the found path (bug-447); any code reusing key_find's rc for success checks must normalise it.
- Controller test binary reply observation: no xce_reply hook exists in src/lib; observe replies on a zeroed no-iomgr endpoint_t — xce_reply endpoint_enq()s onto ep->outq; drain with endpoint_deq + raw capnp read_Request/read_RequestReply (test_commit_barrier_reply.c).
- makeJob needs job_obj_init() once per process AND xce_job_t.submit != NULL; makeString needs xstring_table_init (test_worker_join.c inits it unguarded at group entry — probe makeString for NULL before initialising to stay -t-selection safe).
## Do-Not-Repeat

140
.wolf/memory.md
View file

@ -249,119 +249,13 @@
> Consolidated session (0 actions)
## Session: 2026-06-04 04:46
> Consolidated session (144 actions)
> Consolidated session (0 actions)
## Session: 2026-06-05 18:53
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
> Consolidated session (0 actions)
## Session: 2026-06-05 18:57
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 19:12 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~18 |
| 19:19 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | modified XCE_TESTING() | ~2119 |
| 19:21 | Edited src/controller/ha.c | added 6 condition(s) | ~955 |
| 19:21 | Edited src/controller/ha.h | modified 1() | ~209 |
| 19:21 | Edited src/controller/ha.c | added 2 condition(s) | ~379 |
| 19:22 | Edited src/controller/ha.c | modified if() | ~27 |
| 19:41 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~23 |
| 19:46 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | modified XCE_TESTING() | ~1408 |
| 19:47 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 2→2 lines | ~41 |
| 20:05 | Edited src/controller/ha.c | modified if() | ~348 |
| 20:28 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~22 |
| 20:35 | Edited src/controller/config_repl.c | removed 207 lines | ~418 |
| 20:38 | Edited src/controller/testsuite/test_config_repl.c | modified test_ctrlhello_no_repl_messages_enqueued() | ~1189 |
| 20:38 | Edited src/controller/testsuite/test_config_repl.c | 3→5 lines | ~68 |
| 20:54 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~16 |
| 20:59 | Edited src/controller/testsuite/test_ha_role.c | modified XCE_TESTING() | ~1320 |
| 21:00 | Edited src/controller/testsuite/test_ha_role.c | 5→9 lines | ~135 |
| 21:01 | Edited src/controller/ha.c | added 7 condition(s) | ~846 |
| 21:02 | Edited src/controller/ha.h | modified node() | ~242 |
| 21:02 | Edited src/controller/ha.c | modified XCE_TESTING() | ~338 |
| 21:04 | Edited src/controller/ha.c | modified ha_check_majority_convergence() | ~574 |
| 21:42 | Edited src/controller/ha.h | 1→2 lines | ~34 |
| 21:42 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~15 |
| 21:49 | Edited src/controller/testsuite/test_ha_role.c | added 1 condition(s) | ~1918 |
| 21:52 | Edited src/controller/ha.h | 10000 → 60000 | ~14 |
| 21:52 | Edited src/controller/ha.h | 10000 → 60000 | ~30 |
| 21:53 | Edited src/controller/ha.c | added 1 condition(s) | ~441 |
| 21:53 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | inline fix | ~18 |
| 22:03 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~17 |
| 22:10 | Edited src/controller/config_repl.c | 6→11 lines | ~112 |
| 22:10 | Edited src/controller/config_repl.c | 11→16 lines | ~142 |
| 22:10 | Edited src/controller/config_repl.c | added 1 condition(s) | ~91 |
| 22:26 | Edited src/controller/ha.c | added 2 condition(s) | ~248 |
| 22:36 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~12 |
| 22:39 | Edited src/controller/testsuite/test_ha_role.c | modified test_evaluate_repl_action_stream_wal() | ~447 |
| 22:39 | Edited src/controller/testsuite/test_ha_role.c | 6→8 lines | ~127 |
| 22:39 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~1326 |
| 22:40 | Edited src/controller/testsuite/test_config_repl.c | 5→8 lines | ~123 |
| 22:59 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~17 |
| 22:59 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~15 |
| 23:02 | Edited src/controller/ha.h | 6→11 lines | ~213 |
| 23:03 | Edited src/controller/ha.c | my_lsn() → self_lsn() | ~247 |
| 23:03 | Edited src/controller/ha.c | modified if() | ~242 |
| 23:03 | Edited src/controller/ha.c | modified self_tag() | ~130 |
| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 18→20 lines | ~224 |
| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 9→11 lines | ~112 |
| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | modified XCE_TESTING() | ~418 |
| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 5→7 lines | ~74 |
| 23:05 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 6→9 lines | ~108 |
| 23:05 | Edited src/controller/ha.h | fields() → fix() | ~179 |
| 23:06 | Edited src/controller/testsuite/test_config_repl.c | modified test_snapshot_recv_128kb_two_chunks() | ~1174 |
| 23:07 | Edited src/controller/testsuite/test_config_repl.c | 6→8 lines | ~123 |
| 23:20 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~16 |
| 23:20 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~13 |
| 23:21 | Session end: 54 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 22 reads | ~171678 tok |
| 23:41 | Edited src/controller/ha.c | 2→6 lines | ~118 |
| 23:42 | Edited src/controller/ha.c | 21→23 lines | ~362 |
| 06:50 | task 6.5 integration test: found bug in ha->n_peers (total=3 instead of other=2), fixed, retested — all checks pass | src/controller/ha.c | PASS majority convergence | ~8000 |
| 00:03 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~18 |
| 00:26 | Edited src/controller/ha.c | modified if() | ~144 |
| 00:31 | Edited src/controller/ha.c | modified liveness_poll_fn() | ~133 |
| 00:32 | Edited src/controller/ha.c | modified if() | ~170 |
| 00:36 | Edited src/controller/ha.c | modified XCE_TESTING() | ~165 |
| 01:02 | Edited src/controller/ha.c | 3→2 lines | ~38 |
| 01:06 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~18 |
| 01:26 | Edited src/controller/ha.c | added 4 condition(s) | ~1172 |
| 01:27 | Edited src/controller/ha.c | 13→8 lines | ~164 |
| 01:33 | Session end: 65 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~360703 tok |
| 01:37 | Edited src/controller/ha.c | count() → size() | ~102 |
| 01:37 | Edited src/controller/ha.c | 2→2 lines | ~31 |
| 01:37 | Edited src/controller/ha.c | modified for() | ~124 |
| 01:37 | Edited src/controller/ha.c | modified for() | ~162 |
| 01:38 | Edited src/controller/ha.c | modified for() | ~131 |
| 01:38 | Edited src/controller/ha.h | inline fix | ~29 |
| 01:38 | Edited src/controller/ha.h | modified node() | ~34 |
| 01:39 | Edited src/controller/testsuite/test_ha_role.c | n_peers() → immediately() | ~54 |
| 01:40 | Edited src/controller/testsuite/test_ha_role.c | 2 → 3 | ~17 |
| 01:40 | Edited src/controller/testsuite/test_ha_role.c | 10→10 lines | ~143 |
| 01:41 | Edited src/controller/testsuite/test_ha_role.c | 2 → 3 | ~15 |
| 01:42 | Edited src/controller/testsuite/test_ha_role.c | 7→7 lines | ~71 |
| 01:42 | Edited src/controller/testsuite/test_ha_role.c | 2 → 3 | ~19 |
| 01:42 | Edited src/controller/testsuite/test_ha_role.c | inline fix | ~13 |
| 01:47 | Session end: 79 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~361671 tok |
| 09:55 | E2E cluster lifecycle test (task 6.7) | logs/ctrl-*.log | PASS: build green, RDHUP→election→promotion, config tag preserved, ASAN clean | ~4k |
| 02:56 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~12 |
| 02:57 | Session end: 80 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~361683 tok |
| 03:51 | Session end: 80 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~361683 tok |
| 04:00 | Edited src/controller/ha.c | 8→5 lines | ~49 |
| 04:06 | Edited src/controller/ha.c | 6→8 lines | ~131 |
| 04:06 | Edited src/controller/ha.c | DEBUG2() → INFO() | ~36 |
| 04:28 | Edited src/controller/ha.c | modified if() | ~180 |
| 04:40 | Edited src/controller/ha.c | modified 1() | ~44 |
| 04:41 | Edited src/controller/ha.c | modified no() | ~121 |
| 04:46 | Edited src/controller/ha.c | INFO() → DEBUG2() | ~115 |
| 05:04 | Edited src/controller/ha.c | added 1 condition(s) | ~304 |
| 05:11 | Edited src/controller/ha.c | 2→1 lines | ~20 |
| 05:27 | Edited src/controller/ha.c | added 1 condition(s) | ~72 |
| 05:43 | Edited src/controller/ha.c | modified no() | ~52 |
| 06:02 | Edited src/lib/delay.c | added 5 condition(s) | ~270 |
| 06:18 | Edited src/controller/ha.c | reduced (-6 lines) | ~32 |
| 06:18 | Edited src/controller/ha.c | modified no() | ~52 |
| 06:28 | Session end: 94 writes across 8 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 35 reads | ~364937 tok |
> Consolidated session (102 actions)
## Session: 2026-06-08 01:03
@ -1003,3 +897,31 @@
| 05:45 | Adversarial review ha-commit-barrier-rollout task 2.2 (barrier helper + walAck retirement): boundary clean (persist.c+repl.c only), 15/15 group, full census 472 Unity PASS / 0 FAIL (= 465 prior + 7 new), configAck byte-equivalent, LSN-0 absorption commented, ce==NULL asymmetry judged defensible — APPROVED | src/controller/persist.c, src/controller/repl.c | APPROVED | ~30k |
| 22:36 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~21 |
| 22:36 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | modified RESOLVED() | ~148 |
## Session: 2026-06-12 00:02
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
## Session: 2026-06-12 00:39
| Time | Action | File(s) | Outcome | ~Tokens |
|------|--------|---------|---------|--------|
| 01:55 | Created src/controller/testsuite/test_commit_barrier_reply.c | — | ~7971 |
| 01:55 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified cbrp_op_thread_fn() | ~102 |
| 01:56 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 6→5 lines | ~87 |
| 01:56 | Edited src/controller/meson.build | 2→6 lines | ~37 |
| 01:56 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~25 |
| 01:56 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~27 |
| 01:57 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→3 lines | ~15 |
| 01:58 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified notify() | ~392 |
| 02:01 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~115 |
| 02:06 | Edited src/controller/testsuite/test_commit_barrier_reply.c | expanded (+6 lines) | ~132 |
| 02:08 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~200 |
| 02:09 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 3→5 lines | ~89 |
| 02:09 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~217 |
| 02:20 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified notify() | ~218 |
| 02:20 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 46→42 lines | ~673 |
| 09:35 | Task 3.1 ha-commit-barrier-rollout: created RED integration tests for commit-first-then-reply (4 opcodes x withheld/acked/validation) | src/controller/testsuite/test_commit_barrier_reply.c, meson.build, controller_test.c | 484 results, 8 RED fails (new file only), 472 baseline green; discovered key_find rc==1 leak in queue_ctrl/worker_ctrl/job_ctrl (bug-447) | ~90k |
| 03:00 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~21 |
| 03:00 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | 1→3 lines | ~255 |

4
src/controller/meson.build
View file

@ -122,7 +122,11 @@ controller_unit_test_src = [
'testsuite/test_ha_liveness.c',
'testsuite/test_commit_engine.c',
'testsuite/test_commit_barrier_rollout.c',
'testsuite/test_commit_barrier_reply.c',
'commit.c',
'job.c',
'queue.c',
'join.c',
'startup_validation.c',
'worker.c',
'persist.c',

2
src/controller/testsuite/controller_test.c
View file

@ -52,6 +52,7 @@ extern void test_ha_rdhup_integration(void);
extern void test_ha_liveness(void);
extern void test_commit_engine(void);
extern void test_commit_barrier_rollout(void);
extern void test_commit_barrier_reply(void);
void setUp(void) { return; }
@ -173,6 +174,7 @@ int main(int argc, char* argv[])
TEST(test_ha_liveness);
TEST(test_commit_engine);
TEST(test_commit_barrier_rollout);
TEST(test_commit_barrier_reply);
return UNITY_END();
}

800
src/controller/testsuite/test_commit_barrier_reply.c Normal file
View file

@ -0,0 +1,800 @@
/*
* test_commit_barrier_reply.c — RED-phase integration tests for
* commit-first-then-reply per opcode (ha-commit-barrier-rollout task 3.1).
* Requirements 1.1 (success reply only after majority commit), 1.2 (commit
* timeout → -ETIMEDOUT error reply), 1.3 (barrier rejection → error reply,
* never success), 1.4 (no success-interpretable reply pre-commit).
*
* Drives the four request-shaped handlers DIRECTLY on a worker pthread
* (vtask-context stand-in; production dispatch: cluster.c:325-363) against
* a real tmpdir persist_t embedded in a test cluster_t with registered
* persistence hooks, a wired commit engine (n_peers=3), a stub standby
* registry (2 dummy standbys), ACKs driven through repl_on_ack:
* do_jobcontrol_op — not-started-job branch → WAL_JOB_FAIL (SIGKILL)
* do_queuecontrol_op — QUEUE_CLOSE → WAL_QUEUE_CTRL
* do_hostcontrol_op — HOST_CLOSE → WAL_WORKER_CTRL/UPDATE
* handle_worker_join — first-time join → WAL_WORKER_CTRL/JOIN
* (via handle_join — handle_worker_join is static)
*
* Reply observation harness: there is no xce_reply test hook in src/lib
* (out of boundary), so replies are observed where the production send
* path deposits them — xce_reply() encodes the reply Request and
* endpoint_enq()s it onto ep->outq (request.c:334-368, netio.c:796). The
* requester endpoint is a zeroed static endpoint_t with no iomgr:
* endpoint_update_epoll() is a no-op and the zeroed pthread mutexes are
* valid locks on Linux, so packets stay queued. Tests drain via
* endpoint_deq() and parse with the raw capnp reader (capn_init_mem →
* read_Request → read_RequestReply), the raw-packet parse pattern of
* test_config_repl.c — least invasive, zero production changes.
*
* RED expectations against the current reply-before-commit handlers
* (anchors re-verified 2026-06-12: job.c:932, queue.c:130, worker.c:700,
* join.c:107-109):
* - *_withheld_acks_replies_error / *_success_reply_only_after_commit:
* the handlers reply immediately, before any covering ACK → the
* Req 1.1/1.2/1.4 ordering assertions fail. Additional RED finding:
* queue_ctrl/worker_ctrl/job_ctrl leak key_find's found rc (== 1), so
* queue/host control skip their AFTER_EVENT hook (NO WAL entry) and
* reply status 1 — pinned by the append assertion, 3.2 must close it.
* - *_error_reply_immediate: local validation failures already reply
* immediately with their existing codes — contract pins that stay
* green before AND after the task-3.2 restructure.
*
* Fixtures reused from test_commit_barrier_rollout.c (engine + stub
* registry + drop send hook + LSN-0 priming append) and test_worker_join.c
* (kvmgrs, resources, hostattrs, hostconfig; its collect_confvars stub
* also serves this file's join path).
*/
/* rtti.h must come first — it defines 'var' and 'typeinfo_t' used by kvmgr.h */
#include <rtti.h>
#include <errno.h>
#include <pthread.h>
#include <signal.h>
#include <stdatomic.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <capnp_c.h>
#include <unity/unity.h>
#include "controller.capnp.h"
#include "xce/cluster_def.h"
#include "xce/types.h"
#include "cluster.h"
#include "clusterconfig.h"
#include "commit.h"
#include "hook.h"
#include "jobj.h"
#include "kvmgr.h"
#include "myconf.h"
#include "netio.h"
#include "persist.h"
#include "qobj.h"
#include "repl.h"
#include "resobj.h"
#include "request.h"
#include "typemap.h"
#include "wal_types.h"
#include "wobj.h"
#include "xstring.h"
#include "zarray.h"
/* ── Link stubs ──────────────────────────────────────────────────────────
* job.c / join.c reference these cross-module functions (opupd.c, sched.c,
* apiserver.c — not part of the unit-test link). None of them sits on the
* four code paths under test (jobctrl not-started branch, queue/host ctrl,
* worker join); stubbed so the binary links, same precedent as
* test_worker_join.c's collect_confvars stub. */
/* clang-format off */
int notify(cluster_t *c, endpoint_t *e, xce_notify_data_t *d)
{ (void)c; (void)e; (void)d; return 0; }
worker_t *get_ep_worker(cluster_t *c, endpoint_t *e)
{ (void)c; (void)e; return NULL; }
int update_host_runusage(xce_job_runusage_t **r, char *h, int n,
xce_rusage_t **u)
{ (void)r; (void)h; (void)n; (void)u; return 0; }
int update_pids(job_pidinfo_t **i, int n, int *p)
{ (void)i; (void)n; (void)p; return 0; }
int update_pgids(job_pidinfo_t **i, int n, int *p)
{ (void)i; (void)n; (void)p; return 0; }
int on_scheduler_join(cluster_t *c) { (void)c; return 0; }
int add_apiserver(cluster_t *c, char *n, endpoint_t *e)
{ (void)c; (void)n; (void)e; return 0; }
int on_apiserver_join(cluster_t *c, endpoint_t *e)
{ (void)c; (void)e; return 0; }
/* clang-format on */
/* Test-injectable send hook (repl.c, XCE_TESTING) — keeps repl_broadcast
* from dereferencing the dummy standby endpoints. */
extern int (*repl_broadcast_send_hook)(void *ep,
wal_catchup_entry_t **entries,
int n_entries);
static int cbrp_drop_send_hook(void *ep, wal_catchup_entry_t **entries,
int n_entries)
{
(void)ep; (void)entries; (void)n_entries;
return 0; /* pretend success, drop the data */
}
static uint64_t cbrp_mono_ms(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return (uint64_t)ts.tv_sec * 1000u + (uint64_t)ts.tv_nsec / 1000000u;
}
static void cbrp_sleep_ms(uint32_t ms)
{
struct timespec ts;
ts.tv_sec = ms / 1000u;
ts.tv_nsec = (long)(ms % 1000u) * 1000000L;
nanosleep(&ts, NULL);
}
/* ── Reply observation (see file header for the harness rationale) ───── */
typedef struct {
int n_replies; /* replymsg packets seen by this scan */
int32_t last_status; /* status of the last replymsg observed */
uint64_t committed_at_obs; /* committed_lsn sampled at observation time */
} cbrp_reply_obs_t;
static persist_t *g_obs_persist; /* sampled into committed_at_obs */
/* Drain every packet queued on ep->outq; record replymsg status. Non-reply
* packets (e.g. on_worker_join's config push) are parsed and discarded. */
static void cbrp_scan_replies(endpoint_t *ep, cbrp_reply_obs_t *obs)
{
data_item_t *it;
while ((it = endpoint_deq(ep)) != NULL) {
io_packet_t *pkt = ITPTR_TO_PACKET(it);
struct capn c;
struct Request rq;
Request_ptr rp;
capn_init_mem(&c, (const uint8_t *)pkt->data, pkt->sz, 0);
rp.p = capn_getp(capn_root(&c), 0, 1);
read_Request(&rq, rp);
if (rq.body_which == Request_body_replymsg) {
struct RequestReply rr;
read_RequestReply(&rr, rq.body.replymsg);
obs->n_replies += 1;
obs->last_status = rr.status;
obs->committed_at_obs = atomic_load(&g_obs_persist->committed_lsn);
}
capn_free(&c);
free_packet(pkt);
free(pkt);
}
}
/* Poll until at least one replymsg is observed or the deadline expires. */
static void cbrp_poll_reply(endpoint_t *ep, uint32_t deadline_ms,
cbrp_reply_obs_t *obs)
{
uint64_t t0 = cbrp_mono_ms();
for (;;) {
cbrp_scan_replies(ep, obs);
if (obs->n_replies > 0 || cbrp_mono_ms() - t0 >= deadline_ms)
return;
cbrp_sleep_ms(5);
}
}
/* Poll until the WAL grows past lsn_before (the operation's append became
* visible) or the deadline expires; returns the appended watermark
* (current_lsn - 1) or 0 on timeout. */
static uint64_t cbrp_poll_append(persist_t *p, uint64_t lsn_before,
uint32_t deadline_ms)
{
uint64_t t0 = cbrp_mono_ms();
for (;;) {
uint64_t cur;
pthread_mutex_lock(&p->lock);
cur = p->current_lsn;
pthread_mutex_unlock(&p->lock);
if (cur > lsn_before)
return cur - 1;
if (cbrp_mono_ms() - t0 >= deadline_ms)
return 0;
cbrp_sleep_ms(5);
}
}
/* ── Handler-on-a-pthread runner (vtask-context stand-in) ───────────── */
typedef int (*cbrp_op_fn_t)(cluster_t *, endpoint_t *, xce_request_t *);
typedef struct {
cbrp_op_fn_t fn;
xce_request_t *req;
int rc;
} cbrp_op_arg_t;
/* Statics survive Unity's longjmp on assertion failure. */
static cbrp_op_arg_t g_op_arg;
static pthread_t g_op_tid;
static cluster_t g_hc; /* harness cluster (defined below) */
static endpoint_t g_req_ep; /* requester endpoint (capture, defined below) */
static void *cbrp_op_thread_fn(void *arg)
{
cbrp_op_arg_t *a = arg;
a->rc = a->fn(&g_hc, &g_req_ep, a->req);
return NULL;
}
/* ── Cluster harness ─────────────────────────────────────────────────── */
/* Resource names/types required by init_worker → apply_join_state
* (add_hostattrs_resvals, worker.c) — mirror of test_worker_join.c. */
#define CBRP_N_RES 9
static const char *cbrp_res_names[CBRP_N_RES] = {
"type", "model", "cpuf", "maxcpus", "maxmem",
"maxswap", "maxtmp", "ndisks", "slots"
};
static const int cbrp_res_valtypes[CBRP_N_RES] = {
XCE_VAL_STRING, XCE_VAL_STRING, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC,
XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC,
XCE_VAL_NUMERIC
};
/* Statics survive Unity's longjmp on assertion failure.
* (g_hc / g_req_ep are tentatively defined above for the thread runner.) */
static commit_engine_t g_hce; /* wired commit engine */
static int g_hep_a, g_hep_b;/* dummy standby endpoints */
static ep_private_t g_req_priv;
static xce_peerinfo_t g_req_from;
static char g_hc_tmpdir[256];
/* myconf scaffolding for find_matched_hostconfig (worker join path) */
static myconf_t g_hc_myconf;
static cluster_config_t g_hc_clusterconf;
static xce_hostconfig_t g_hc_hconfig;
static xce_hostconfig_t *g_hc_hconfigs[1];
static void cbrp_setup(uint32_t sync_timeout_ms)
{
persist_config_t cfg;
kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING,
.nslots = 64, .delta = 16 };
kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT,
.nslots = 64, .delta = 16 };
int i;
memset(&g_hc, 0, sizeof(g_hc));
memset(&g_req_ep, 0, sizeof(g_req_ep));
memset(&g_req_priv, 0, sizeof(g_req_priv));
memset(&g_req_from, 0, sizeof(g_req_from));
/* Requester endpoint: zeroed, no iomgr — xce_reply packets stay on
* ep->outq (see file header). */
g_req_ep.fd = -1;
g_req_ep.data = &g_req_priv;
g_req_from.name = (char *)"requester";
g_req_from.key = (char *)"";
g_hc.myhost = &g_hc.myhost_info;
g_hc.myhost_info.name = (char *)"ctrl-test";
g_hc.myhost_info.key = (char *)"";
/* ha zero-init: CTRL_PRIMARY == 0 — handle_join's standby gates pass */
TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&g_hc.hooksys,
XCE_CONTROLLER_HOOK_MAX));
g_hc.jobmgr = makeInstance(&xkvmgr_type, &int_params);
g_hc.qmgr = makeInstance(&xkvmgr_type, &str_params);
g_hc.workermgr = makeInstance(&xkvmgr_type, &str_params);
g_hc.resmgr = makeInstance(&xkvmgr_type, &str_params);
TEST_ASSERT_NOT_NULL(g_hc.jobmgr);
TEST_ASSERT_NOT_NULL(g_hc.qmgr);
TEST_ASSERT_NOT_NULL(g_hc.workermgr);
TEST_ASSERT_NOT_NULL(g_hc.resmgr);
zarray_init(&g_hc.sharedinst, 64, NULL);
for (i = 0; i < CBRP_N_RES; i++) {
xce_resource_t rdef;
memset(&rdef, 0, sizeof(rdef));
rdef.name = (char *)cbrp_res_names[i];
rdef.valtype = cbrp_res_valtypes[i];
rdef.idx = -1;
TEST_ASSERT_NOT_NULL_MESSAGE(makeResource(g_hc.resmgr, &rdef),
cbrp_res_names[i]);
}
/* hostconfig "default" so find_matched_hostconfig succeeds for joins */
memset(&g_hc_myconf, 0, sizeof(g_hc_myconf));
memset(&g_hc_clusterconf, 0, sizeof(g_hc_clusterconf));
memset(&g_hc_hconfig, 0, sizeof(g_hc_hconfig));
g_hc_hconfig.name = (char *)"default";
g_hc_hconfig.hosttype = (char *)"!";
g_hc_hconfig.hostmodel = (char *)"!";
g_hc_hconfig.mxj = 2;
g_hc_hconfigs[0] = &g_hc_hconfig;
g_hc_clusterconf.nhconfigs = 1;
g_hc_clusterconf.hconfigs = g_hc_hconfigs;
g_hc_myconf.cluster_config = &g_hc_clusterconf;
g_hc.myconf = &g_hc_myconf;
/* Real persist over a private tmpdir; cluster back-pointer set by
* persist_init so persist_start registers the production hooks. */
cfg = persist_config_defaults();
cfg.wal_sync_ms = 5000;
cfg.sync_timeout_ms = sync_timeout_ms;
snprintf(g_hc_tmpdir, sizeof(g_hc_tmpdir), "/tmp/xce_cbrp31_XXXXXX");
TEST_ASSERT_NOT_NULL(mkdtemp(g_hc_tmpdir));
persist_set_workdir(&g_hc.persist, g_hc_tmpdir, "ctrl");
TEST_ASSERT_EQUAL_INT(0, persist_init(&g_hc.persist, &cfg, &g_hc));
TEST_ASSERT_EQUAL_INT(0, persist_start(&g_hc.persist));
/* 3-member cluster: 2 stub standbys, engine wired, ACKs withheld
* until a test explicitly delivers them via repl_on_ack. */
TEST_ASSERT_TRUE(
repl_add_standby(&g_hc.persist.repl, &g_hep_a, "sb_a", 7400) >= 0);
TEST_ASSERT_TRUE(
repl_add_standby(&g_hc.persist.repl, &g_hep_b, "sb_b", 7401) >= 0);
TEST_ASSERT_EQUAL_INT(0, commit_engine_init(&g_hce, &g_hc.persist, 3, 8));
repl_set_commit_engine(&g_hc.persist.repl, &g_hce);
repl_broadcast_send_hook = cbrp_drop_send_hook;
g_obs_persist = &g_hc.persist;
/* Priming append so operation entries land at LSN >= 1 (the engine
* rejects lsn == 0 — accepted interim LSN-0 edge, ha-commit-engine). */
TEST_ASSERT_EQUAL_INT(
0, persist_wal_async(&g_hc.persist, WAL_JOB_UPDATE, "p", 1));
TEST_ASSERT_EQUAL_UINT64(1, g_hc.persist.current_lsn);
}
static void cbrp_teardown(void)
{
cbrp_reply_obs_t drain = { 0 };
cbrp_scan_replies(&g_req_ep, &drain); /* free any leftover packets */
repl_broadcast_send_hook = NULL;
commit_engine_fini(&g_hce);
persist_stop(&g_hc.persist);
persist_fini(&g_hc.persist);
zarray_fini(&g_hc.sharedinst);
if (g_hc.resmgr) dereference(g_hc.resmgr);
if (g_hc.workermgr) dereference(g_hc.workermgr);
if (g_hc.qmgr) dereference(g_hc.qmgr);
if (g_hc.jobmgr) dereference(g_hc.jobmgr);
xce_hook_fini(&g_hc.hooksys);
}
/* ── Per-opcode entity + request builders ────────────────────────────── */
/* job 42, submitted but never started (allocs == NULL): SIGKILL takes the
* not-started branch of job_ctrl → mark_job_fail → WAL_JOB_FAIL. */
static xce_signal_t g_sigkill = { .signo = SIGKILL, .force = 0 };
static xce_jobctrl_t g_jobctrl;
static xce_request_t g_jobctrl_req;
static job_submit_data_t g_job_submit; /* job_init requires submit != NULL */
static void cbrp_arrange_jobctrl(uint64_t jobid, int create_job)
{
if (create_job) {
xce_job_t jdef;
static int jobobj_ready; /* jobj module tables — once per process */
if (!jobobj_ready) {
TEST_ASSERT_EQUAL_INT(0, job_obj_init(64, 64));
jobobj_ready = 1;
}
memset(&g_job_submit, 0, sizeof(g_job_submit));
memset(&jdef, 0, sizeof(jdef));
jdef.jobid = jobid;
jdef.state = JOB_STAT_PEND;
jdef.submit = &g_job_submit;
TEST_ASSERT_NOT_NULL(makeJob(g_hc.jobmgr, &jdef));
}
memset(&g_jobctrl, 0, sizeof(g_jobctrl));
g_jobctrl.action = JobControl_u_siginfo;
g_jobctrl.jobid = jobid;
g_jobctrl.u.siginfo = &g_sigkill;
memset(&g_jobctrl_req, 0, sizeof(g_jobctrl_req));
g_jobctrl_req.opcode = OPCode_jobcontrol;
g_jobctrl_req.seq = 11;
g_jobctrl_req.from = &g_req_from;
g_jobctrl_req.body_which = Request_body_jobctrlmsg;
g_jobctrl_req.body.jobctrlmsg = &g_jobctrl;
}
/* queue "q1": QUEUE_CLOSE → status |= QUEUE_STAT_CLOSED → WAL_QUEUE_CTRL */
static xce_queuectrl_t g_queuectrl;
static xce_request_t g_queuectrl_req;
static void cbrp_arrange_queuectrl(const char *qname, int create_queue)
{
if (create_queue) {
xce_queue_t qdef;
memset(&qdef, 0, sizeof(qdef));
qdef.queue = (char *)qname;
TEST_ASSERT_NOT_NULL(makeQueue(g_hc.qmgr, &qdef));
}
memset(&g_queuectrl, 0, sizeof(g_queuectrl));
g_queuectrl.action = QUEUE_CLOSE;
g_queuectrl.queue = (char *)qname;
memset(&g_queuectrl_req, 0, sizeof(g_queuectrl_req));
g_queuectrl_req.opcode = OPCode_queuecontrol;
g_queuectrl_req.seq = 12;
g_queuectrl_req.from = &g_req_from;
g_queuectrl_req.body_which = Request_body_queuectrlmsg;
g_queuectrl_req.body.queuectrlmsg = &g_queuectrl;
}
/* worker "hostA": HOST_CLOSE → status |= XCE_WORKER_CLOSED →
* WAL_WORKER_CTRL detail WORKER_UPDATE */
static xce_hostctrl_t g_hostctrl;
static xce_request_t g_hostctrl_req;
static void cbrp_arrange_hostctrl(const char *host, int create_worker)
{
if (create_worker) {
xce_worker_t wdef;
hostattrs_t *attrs = calloc(1, sizeof(*attrs));
TEST_ASSERT_NOT_NULL(attrs);
attrs->hosttype = makeString("LINUX");
attrs->hostmodel = makeString("x86");
attrs->cpufactor = 1.0f;
attrs->maxcpus = 4;
memset(&wdef, 0, sizeof(wdef));
wdef.name = (char *)host;
wdef.attrs = attrs;
TEST_ASSERT_NOT_NULL(makeWorker(g_hc.workermgr, &wdef));
}
memset(&g_hostctrl, 0, sizeof(g_hostctrl));
g_hostctrl.action = HOST_CLOSE;
g_hostctrl.hostname = (char *)host;
memset(&g_hostctrl_req, 0, sizeof(g_hostctrl_req));
g_hostctrl_req.opcode = OPCode_hostcontrol;
g_hostctrl_req.seq = 13;
g_hostctrl_req.from = &g_req_from;
g_hostctrl_req.body_which = Request_body_hostctrlmsg;
g_hostctrl_req.body.hostctrlmsg = &g_hostctrl;
}
/* worker join: handle_join with role HostRole_worker reaches the static
* handle_worker_join; first-time join → init_worker → AFTER_EVENT hook →
* WAL_WORKER_CTRL detail WORKER_JOIN. */
static xce_joindata_t g_joindata;
static hostattrs_t g_join_attrs;
static xce_peerinfo_t g_join_from;
static xce_request_t g_join_req;
static void cbrp_arrange_worker_join(const char *host)
{
memset(&g_join_attrs, 0, sizeof(g_join_attrs));
g_join_attrs.hosttype = makeString("LINUX");
g_join_attrs.hostmodel = makeString("x86");
g_join_attrs.cpufactor = 1.0f;
g_join_attrs.maxcpus = 4;
g_join_attrs.maxmem = 16;
g_join_attrs.maxswap = 32;
g_join_attrs.maxtmp = 64;
g_join_attrs.ndisks = 1;
memset(&g_joindata, 0, sizeof(g_joindata));
g_joindata.role = HostRole_worker;
g_joindata.data_which = JoinData_data_attrs;
g_joindata.data.attrs = &g_join_attrs;
memset(&g_join_from, 0, sizeof(g_join_from));
g_join_from.name = (char *)host;
g_join_from.key = (char *)"";
memset(&g_join_req, 0, sizeof(g_join_req));
g_join_req.opcode = OPCode_join;
g_join_req.seq = 14;
g_join_req.from = &g_join_from;
g_join_req.body_which = Request_body_joinmsg;
g_join_req.body.joinmsg = &g_joindata;
}
/* ── Shared case drivers ─────────────────────────────────────────────── */
/* Withheld-ACK case (Req 1.2, 1.4): the requester reply must carry
* -ETIMEDOUT and no success reply may be observable while the operation's
* entry exceeds committed_lsn.
* RED: the current handlers reply SUCCESS immediately. */
static void cbrp_drive_withheld(cbrp_op_fn_t fn, xce_request_t *req,
uint32_t sync_timeout_ms)
{
cbrp_reply_obs_t obs = { 0 };
uint64_t lsn_before, watermark, committed_final;
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
/* Act: handler on a worker pthread (post-3.2 it blocks at the barrier) */
g_op_arg.fn = fn;
g_op_arg.req = req;
g_op_arg.rc = -9999;
TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL,
cbrp_op_thread_fn, &g_op_arg));
/* The reply may legitimately take up to the commit timeout */
cbrp_poll_reply(&g_req_ep, sync_timeout_ms + 1500, &obs);
pthread_join(g_op_tid, NULL);
watermark = cbrp_poll_append(&g_hc.persist, lsn_before, 1000);
committed_final = atomic_load(&g_hc.persist.committed_lsn);
/* Assert: the operation must produce its WAL entry — nothing to
* commit-confirm otherwise (Req 1.1). RED finding: queue_ctrl and
* worker_ctrl leak key_find's found rc (== 1) into the `if (rc == 0)`
* hook gate → NO WAL entry, status-1 reply (header; 3.2 closes it). */
TEST_ASSERT_TRUE_MESSAGE(watermark > 0,
"request-shaped control op must append its WAL entry (queue_ctrl/"
"worker_ctrl currently skip the AFTER_EVENT hook: rc==1 leak)");
/* ...which never committed (ACKs withheld) */
TEST_ASSERT_TRUE_MESSAGE(committed_final < watermark,
"committed_lsn must stay below the entry with ACKs withheld");
/* ...the requester got exactly an error reply */
TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0,
"the requester must receive a reply");
TEST_ASSERT_TRUE_MESSAGE(
!(obs.last_status == 0 && obs.committed_at_obs < watermark),
"no success reply may be observable while the operation's entry "
"exceeds committed_lsn (Req 1.4)");
TEST_ASSERT_EQUAL_INT_MESSAGE(-ETIMEDOUT, obs.last_status,
"withheld ACKs past the commit timeout must surface -ETIMEDOUT "
"to the requester (Req 1.2)");
}
/* Covering-ACK case (Req 1.1, 1.4): the success reply must be observable
* only after committed_lsn covers the operation's entry.
* RED: the current handlers reply success before any ACK arrives. */
static void cbrp_drive_acked(cbrp_op_fn_t fn, xce_request_t *req)
{
cbrp_reply_obs_t early = { 0 };
cbrp_reply_obs_t final = { 0 };
uint64_t lsn_before, watermark;
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
g_op_arg.fn = fn;
g_op_arg.req = req;
g_op_arg.rc = -9999;
TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL,
cbrp_op_thread_fn, &g_op_arg));
/* Wait until the operation's append is visible, then hold the handler
* blocked for a beat and sample the reply capture BEFORE any ACK. */
watermark = cbrp_poll_append(&g_hc.persist, lsn_before, 2000);
cbrp_sleep_ms(150);
cbrp_scan_replies(&g_req_ep, &early);
/* Deliver the covering ACK, then collect the (now permitted) reply */
repl_on_ack(&g_hc.persist.repl, &g_hep_a, watermark);
cbrp_poll_reply(&g_req_ep, 2000, &final);
pthread_join(g_op_tid, NULL);
/* Assert: the operation must produce its WAL entry (see the rc==1
* leak note in cbrp_drive_withheld — same RED finding applies) */
TEST_ASSERT_TRUE_MESSAGE(watermark > 0,
"request-shaped control op must append its WAL entry (queue_ctrl/"
"worker_ctrl currently skip the AFTER_EVENT hook: rc==1 leak)");
/* THE ordering gate: nothing the requester can read as confirmation
* may exist before the covering ACK (Req 1.1, 1.4) */
TEST_ASSERT_EQUAL_INT_MESSAGE(0, early.n_replies,
"no reply may be observable before committed_lsn covers the "
"operation's entry (Req 1.1, 1.4)");
/* After the ACK: success reply, entry committed */
TEST_ASSERT_TRUE_MESSAGE(final.n_replies > 0,
"the requester must receive the reply once the entry commits");
TEST_ASSERT_EQUAL_INT_MESSAGE(0, final.last_status,
"the post-commit reply must report success (Req 1.1)");
TEST_ASSERT_TRUE_MESSAGE(final.committed_at_obs >= watermark,
"the success reply must be observable only with committed_lsn "
"covering the operation's entry (Req 1.1)");
}
/* Local-validation-failure pin (Req 1.4 boundary, task brief): the
* existing error reply stays immediate — no barrier wait, no WAL entry.
* Green before AND after the task-3.2 restructure (latency-bounded). */
static void cbrp_drive_validation_error(cbrp_op_fn_t fn, xce_request_t *req,
int32_t expect_status,
int check_exact_status)
{
cbrp_reply_obs_t obs = { 0 };
uint64_t lsn_before, lsn_after, t0, elapsed_ms;
pthread_mutex_lock(&g_hc.persist.lock);
lsn_before = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
/* Act: synchronous — a validation failure must never block */
t0 = cbrp_mono_ms();
(void)fn(&g_hc, &g_req_ep, req);
elapsed_ms = cbrp_mono_ms() - t0;
cbrp_scan_replies(&g_req_ep, &obs);
pthread_mutex_lock(&g_hc.persist.lock);
lsn_after = g_hc.persist.current_lsn;
pthread_mutex_unlock(&g_hc.persist.lock);
/* Assert: immediate error reply with the existing code, no append */
TEST_ASSERT_TRUE_MESSAGE(elapsed_ms < 1000,
"local validation failures must reply without a barrier wait");
TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0,
"validation failure must still produce a requester reply");
if (check_exact_status)
TEST_ASSERT_EQUAL_INT_MESSAGE(expect_status, obs.last_status,
"validation failure must keep its existing error code");
else
TEST_ASSERT_TRUE_MESSAGE(obs.last_status < 0,
"validation failure must reply with an error status");
TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before, lsn_after,
"a rejected operation must append no WAL entry");
}
/* ── do_jobcontrol_op (not-started branch → WAL_JOB_FAIL) ────────────── */
static void test_jobctrl_withheld_acks_replies_error(void)
{
cbrp_setup(300);
cbrp_arrange_jobctrl(42, 1);
cbrp_drive_withheld(do_jobcontrol_op, &g_jobctrl_req, 300);
cbrp_teardown();
printf("test_jobctrl_withheld_acks_replies_error: PASS\n");
}
static void test_jobctrl_success_reply_only_after_commit(void)
{
cbrp_setup(1500);
cbrp_arrange_jobctrl(43, 1);
cbrp_drive_acked(do_jobcontrol_op, &g_jobctrl_req);
cbrp_teardown();
printf("test_jobctrl_success_reply_only_after_commit: PASS\n");
}
static void test_jobctrl_unknown_job_error_reply_immediate(void)
{
cbrp_setup(3000);
cbrp_arrange_jobctrl(9999, 0); /* job does not exist */
cbrp_drive_validation_error(do_jobcontrol_op, &g_jobctrl_req,
-ENOENT, 1);
cbrp_teardown();
printf("test_jobctrl_unknown_job_error_reply_immediate: PASS\n");
}
/* ── do_queuecontrol_op (QUEUE_CLOSE → WAL_QUEUE_CTRL) ───────────────── */
static void test_queuectrl_withheld_acks_replies_error(void)
{
cbrp_setup(300);
cbrp_arrange_queuectrl("q1", 1);
cbrp_drive_withheld(do_queuecontrol_op, &g_queuectrl_req, 300);
cbrp_teardown();
printf("test_queuectrl_withheld_acks_replies_error: PASS\n");
}
static void test_queuectrl_success_reply_only_after_commit(void)
{
cbrp_setup(1500);
cbrp_arrange_queuectrl("q2", 1);
cbrp_drive_acked(do_queuecontrol_op, &g_queuectrl_req);
cbrp_teardown();
printf("test_queuectrl_success_reply_only_after_commit: PASS\n");
}
static void test_queuectrl_unknown_queue_error_reply_immediate(void)
{
cbrp_setup(3000);
cbrp_arrange_queuectrl("no-such-queue", 0);
cbrp_drive_validation_error(do_queuecontrol_op, &g_queuectrl_req,
-ENOENT, 1);
cbrp_teardown();
printf("test_queuectrl_unknown_queue_error_reply_immediate: PASS\n");
}
/* ── do_hostcontrol_op (HOST_CLOSE → WAL_WORKER_CTRL/UPDATE) ─────────── */
static void test_hostctrl_withheld_acks_replies_error(void)
{
cbrp_setup(300);
cbrp_arrange_hostctrl("hostA", 1);
cbrp_drive_withheld(do_hostcontrol_op, &g_hostctrl_req, 300);
cbrp_teardown();
printf("test_hostctrl_withheld_acks_replies_error: PASS\n");
}
static void test_hostctrl_success_reply_only_after_commit(void)
{
cbrp_setup(1500);
cbrp_arrange_hostctrl("hostB", 1);
cbrp_drive_acked(do_hostcontrol_op, &g_hostctrl_req);
cbrp_teardown();
printf("test_hostctrl_success_reply_only_after_commit: PASS\n");
}
static void test_hostctrl_unknown_host_error_reply_immediate(void)
{
cbrp_setup(3000);
cbrp_arrange_hostctrl("no-such-host", 0);
cbrp_drive_validation_error(do_hostcontrol_op, &g_hostctrl_req,
-ENOENT, 1);
cbrp_teardown();
printf("test_hostctrl_unknown_host_error_reply_immediate: PASS\n");
}
/* ── handle_worker_join via handle_join (→ WAL_WORKER_CTRL/JOIN) ─────── */
static void test_worker_join_withheld_acks_replies_error(void)
{
cbrp_setup(300);
cbrp_arrange_worker_join("workerA");
cbrp_drive_withheld(handle_join, &g_join_req, 300);
cbrp_teardown();
printf("test_worker_join_withheld_acks_replies_error: PASS\n");
}
static void test_worker_join_success_reply_only_after_commit(void)
{
cbrp_setup(1500);
cbrp_arrange_worker_join("workerB");
cbrp_drive_acked(handle_join, &g_join_req);
cbrp_teardown();
printf("test_worker_join_success_reply_only_after_commit: PASS\n");
}
static void test_worker_join_no_hostconfig_error_reply_immediate(void)
{
cbrp_setup(3000);
cbrp_arrange_worker_join("workerC");
/* Remove every hostconfig: find_matched_hostconfig returns NULL and
* handle_worker_join must reply its existing error immediately —
* the (future) barrier sits between the AFTER_EVENT hook and the
* SUCCESS reply only; error replies stay immediate. */
g_hc_clusterconf.nhconfigs = 0;
g_hc_clusterconf.hconfigs = NULL;
cbrp_drive_validation_error(handle_join, &g_join_req, 0, 0);
cbrp_teardown();
printf("test_worker_join_no_hostconfig_error_reply_immediate: PASS\n");
}
/* ── Test group entry point ──────────────────────────────────────────── */
void test_commit_barrier_reply(void)
{
/* makeString needs the global xstring table. In the full suite,
* test_worker_join already initialised it (unguarded, its line 520);
* standalone (-t test_commit_barrier_reply) it is uninitialised and
* makeString returns NULL. Probe instead of double-initialising. */
if (makeString((char *)"cbrp-probe") == NULL)
xstring_table_init(139);
UNITY_BEGIN();
/* do_jobcontrol_op — Req 1.1, 1.2, 1.4 */
RUN_TEST(test_jobctrl_withheld_acks_replies_error);
RUN_TEST(test_jobctrl_success_reply_only_after_commit);
RUN_TEST(test_jobctrl_unknown_job_error_reply_immediate);
/* do_queuecontrol_op */
RUN_TEST(test_queuectrl_withheld_acks_replies_error);
RUN_TEST(test_queuectrl_success_reply_only_after_commit);
RUN_TEST(test_queuectrl_unknown_queue_error_reply_immediate);
/* do_hostcontrol_op */
RUN_TEST(test_hostctrl_withheld_acks_replies_error);
RUN_TEST(test_hostctrl_success_reply_only_after_commit);
RUN_TEST(test_hostctrl_unknown_host_error_reply_immediate);
/* handle_worker_join (via handle_join) */
RUN_TEST(test_worker_join_withheld_acks_replies_error);
RUN_TEST(test_worker_join_success_reply_only_after_commit);
RUN_TEST(test_worker_join_no_hostconfig_error_reply_immediate);
UNITY_END();
}