diff --git a/.kiro/specs/ha-commit-barrier-rollout/tasks.md b/.kiro/specs/ha-commit-barrier-rollout/tasks.md index 2de14f9..6225175 100644 --- a/.kiro/specs/ha-commit-barrier-rollout/tasks.md +++ b/.kiro/specs/ha-commit-barrier-rollout/tasks.md @@ -32,7 +32,7 @@ - _Requirements: 1.1, 1.2, 1.3_ - [ ] 3. Reply-ordering restructure of request-shaped handlers -- [ ] 3.1 Write RED integration tests for commit-first-then-reply per opcode +- [x] 3.1 Write RED integration tests for commit-first-then-reply per opcode - Tests drive the four handlers directly (vtask-context test harness, stub standby registry): for each of `do_jobcontrol_op` (not-started-job branch, producing WAL_JOB_UPDATE/WAL_JOB_FAIL), `do_queuecontrol_op`, `do_hostcontrol_op`, `handle_worker_join` — with ACKs withheld the requester reply carries an error (`-ETIMEDOUT`) and no success reply is ever observable while the operation's entries exceed `committed_lsn`; with ACKs supplied the success reply is observable only after `committed_lsn` covers the entries; local validation failures still reply immediately with their existing error codes - Observable completion: new cases FAIL (red) against the current reply-before-commit handlers (anchors: job.c:932, queue.c:130, worker.c:700, join.c:107-109 — re-verify with grep) - _Requirements: 1.1, 1.2, 1.3, 1.4_ @@ -112,5 +112,7 @@ - (1.1) wal_entry_type has 13 values (wal_types.h:44-58), no WAL_NOOP yet (lands in ha-single-leader-role, which must add a classification row + sweep entry — the totality sweep will catch it). - (1.1) Suite baseline is ZERO failures since ha-commit-engine 6.1. detect_leaks=0 remains the harness norm (6 inventoried pre-existing leak groups). - (1.2) CENSUS METHOD: only ~9 of 48 groups emit Unity summary blocks; the rest report per-test PASS lines only — count distinct Unity results (~465) and grep FAIL lines, never sum summary blocks. commit_submit(RETAIN) early-returns BEFORE taking ce->lock (commit.c:233-239) — registration is lock-free, safe with entity write locks held. wal_hook_append_classified is the single classified-append seam; appended_lsn read = current_lsn - 1 under p->lock (benign race, commented). +- (3.1) **bug-447 CONFIRMED (reviewer-verified to librtti disassembly)**: key_find returns 1 on found (unnormalized from xkvmgr_find/kvtab_find); queue_ctrl (queue.c:61, gate :106) and worker_ctrl (worker.c:644, gate :678) leak rc=1 → AFTER_EVENT never fires on the found path → **queue/host control silently unpersisted+unreplicated in production today**; job_ctrl appends via mark_job_* hooks but still replies status 1. Fix point = 3.2's handler rc normalization (key_find is in binary librtti, out of repo). RED tests pin it. +- (3.1) Test link gained job.c/queue.c/join.c (test-only meson change) + 8 inert link stubs in test_commit_barrier_reply.c (none on tested paths — reviewer-traced). Reply capture = zeroed stack endpoint ep->outq inspection (test_config_repl.c:2210 precedent; endpoint_update_epoll no-ops when iomgr NULL). RED-window orphaned fsync threads self-resolve at GREEN (reviewer finding 5). - (2.2) Legacy walAck counting RETIRED: walAck track = entry_lsn==0 → 0 (LSN-0 interim) else unconditional commit_submit (ce==NULL → -EINVAL, fail-loud). configAck track byte-equivalent (task 5's surface intact, incl. its n_active==0 shortcut and acked_config_lsn>0 counting — both die at task 5). persist_commit_barrier: ce==NULL → 0 (vacuous, documented). test_repl_unit's two lsn=0 walAck callers are now vacuous for quorum counting (INFO for task 5/7.2). CENSUS = 472 (465 at 1.2 + 7 from 2.1). - (2.1) BARRIER WATERMARK RESOLVED (reviewer-judged): design's literal "Snapshot lsn = p->current_lsn" is loose phrasing contradicted by its own anchors — the coherent contract is watermark = current_lsn - 1 (last appended; persist.h:71 invariant committed_lsn <= current_lsn - 1 makes the literal reading unsatisfiable). Tests pin current_lsn - 1 with off-by-one killing power in BOTH directions; task 2.2 must implement that. Resolved contract recorded in persist.h doc comment. diff --git a/.wolf/anatomy.md b/.wolf/anatomy.md index 74b2a86..690d92a 100644 --- a/.wolf/anatomy.md +++ b/.wolf/anatomy.md @@ -1,7 +1,7 @@ # anatomy.md -> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T04:13:06.312Z -> Files: 511 tracked | Anatomy hits: 0 | Misses: 0 +> Auto-maintained by OpenWolf. Last scanned: 2026-06-12T10:00:34.785Z +> Files: 509 tracked | Anatomy hits: 0 | Misses: 0 ## ./ @@ -187,7 +187,7 @@ - `.gitignore` — Git ignore rules (~47 tok) - `codegraph.db-shm` (~8738 tok) -- `daemon.log` (~3298 tok) +- `daemon.log` (~3580 tok) - `daemon.pid` (~44 tok) ## .kiro/settings/rules/ @@ -253,7 +253,7 @@ - `requirements.md` — Requirements Document (~3181 tok) - `research.md` — Research & Design Decisions: ha-commit-barrier-rollout (~3672 tok) - `spec.json` (~143 tok) -- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~3743 tok) +- `tasks.md` — Implementation Plan: ha-commit-barrier-rollout (~4337 tok) ## .kiro/specs/ha-commit-engine/ @@ -261,7 +261,7 @@ - `design.md` — Design: ha-commit-engine (~7600 tok) - `requirements.md` — Requirements Document (~2715 tok) - `research.md` — Research Log: ha-commit-engine (~1345 tok) -- `spec.json` (~140 tok) +- `spec.json` (~255 tok) - `tasks.md` — Implementation Plan: ha-commit-engine (~4593 tok) ## .kiro/specs/ha-config-bootstrap/ @@ -393,14 +393,14 @@ ## graphify-out/ -- `.graphify_labels.json` (~5614 tok) +- `.graphify_labels.json` (~5762 tok) - `.graphify_root` (~1 tok) -- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-12) (~45713 tok) -- `manifest.json` (~35154 tok) +- `GRAPH_REPORT.md` — Graph Report - persistent_ha (2026-06-12) (~46969 tok) +- `manifest.json` (~35204 tok) ## graphify-out/cache/ -- `stat-index.json` (~44628 tok) +- `stat-index.json` (~44692 tok) ## graphify-out/cache/ast/ @@ -428,6 +428,7 @@ - `053929f384ca0a4fdabda00415d3f0e0a7f8ccbd86f273deadd54ac0590492a2.json` (~18832 tok) - `0550909cf1a1da30dee6c2653f79809399374ee35919a467be9583fa7d27f5c5.json` (~71978 tok) - `057d67f05183f6fe7fd3414cb6f46de234ebe58fa3dbca7dddc5441ee3f8a2f1.json` (~6218 tok) +- `061bfdbdc03ae56a916d366f3200ca2c610d9ad141e11682e47243197d93632b.json` (~592 tok) - `06578221cb7a75ee06e0460d5a42c48037e3ddcebbc21fae45825886ceaa0355.json` (~15340 tok) - `06a0b286e4c22fb4c981239820aea2042950d1d21dc96de40226b14cc696065e.json` (~8244 tok) - `07014154947b85f86499894c59be83a316bf3fc006c8fc55c316cb893462977e.json` (~3083 tok) @@ -478,6 +479,7 @@ - `151a099eec558a9e4c89bc08aae88ba096681971be1a94c7625343fd9a3a88e5.json` (~7640 tok) - `15224225923ce6697e098312e1dc8429b2768b415534886f344c212446813d83.json` (~495 tok) - `152d9c2445a8247d38b52f6ecab0cf6f36ba144aeb93e0bc7603445e7ed2c6cc.json` (~5075 tok) +- `155212874dafa331b4fd417f7ebd9bafae70754b134e49c5cfb694a47c21def5.json` (~81 tok) - `157041af18754d4729c7ead7ed02190c14fa052b3c65deac29309014894e67a8.json` (~10023 tok) - `157c7929f5a2274e2f98c623d21343ab2f01711d4fd4e742caef92e2bf2b3463.json` (~4857 tok) - `15cc2d188fbb7c7bc937a39311efdc8cd2536b01947309e45f5563c0e40a7923.json` (~1376 tok) @@ -505,10 +507,13 @@ - `1c86ed59f0918f5ea0f47f54917854eb2ce2a9b9b6724a107c9237ed0a7f7ea2.json` (~2916 tok) - `1c906780916fd91e042adaa801f385dc55bc1f29c851908ad61f8c5db0040107.json` (~2163 tok) - `1c9619334c216e67eec7d7a1c420d26476001173e926088e46d4d2acdd52f8d4.json` (~320 tok) +- `1d7c4a75171c44fea2a337b2b8756a0b4aefd3407e5a0cce07089b2722f4939d.json` (~7794 tok) +- `1dcf7b97c92f5149c264d18f36fb408c96372770c52491ec5d320b4a17533612.json` (~8110 tok) - `1e8313e77c1ec1f11b656b60fdcefe81f0d61322c4cdcbf51cf918c7e7965420.json` (~15340 tok) - `1ef77ec32c809ea9f6abe455804f5630dc3a9693e633b03068f3546dfeeab099.json` (~3149 tok) - `1f6c6535ebbdcedd95bf44e6b96f07f1234d7ea54cb747d6e113f90ccf4bb7ea.json` (~13910 tok) - `1f7e683af7e73fefdcf14ce40c2ee81dfa8aa113b7aa90b51067d870710baa7a.json` (~332 tok) +- `1f9c1094d37b3b77b52d3863f257184175c1f040ed46cfbd75b54598f428d466.json` (~81 tok) - `1fdd469bb6d51958ab2ba15266b745abf7014618c2055cf5b098276ea4a501bd.json` (~62630 tok) - `1fe907805f188b6cf4e490bbab545a31e875b83a02bb0e7220e1b8666ddee3e6.json` (~332 tok) - `20456a4402821d8eeb5ccfb2b27a16965b9d4ed85058de966a1f4894fe8443eb.json` (~9852 tok) @@ -526,6 +531,7 @@ - `23ae45376874ad36b52972b9b9031b66aa76f34d1de4b793801908d2dc0bdbc4.json` (~4128 tok) - `23b8085ed43aaaed2d876c21ecb494fa28af40a1cefaffcf5ffc4b131d9ed3f8.json` (~1662 tok) - `23cbeb01ae9cd9669fae81ddb2b2d6c7f89abd17b12365a5f6b256a32a717616.json` (~490 tok) +- `24272ebb7e242bd439e42b571de9c4b67ba0f5a3a09c07bde630b6050cbeef4a.json` (~15340 tok) - `242faeda0abe77b1e66a515917a267528fe2d1b0c091dbced56e493e9cf69737.json` (~3934 tok) - `24fa3f5b3b5b6d59088387aad4130a5e2d657f88706205d6dbe57e8afda2e425.json` (~722 tok) - `251ce70fbb55f062faef6259731b4c82238f4f7af22d2c1ed266c86c15263b99.json` (~502 tok) @@ -545,6 +551,7 @@ - `2b0b0f1eb49df36a2e8fd4f88317124dd1edf9a0f08545a1a53accd6b47b16b1.json` (~309 tok) - `2b9e4e051f5e39e6df4dc1c1ff7611d9a3590262f41b88e978fa664c596524b8.json` (~269 tok) - `2baff6d343f2ec98cec20237f085bb8804623059aa145279f05e1d921aa1f8bc.json` (~68072 tok) +- `2bc3ed1bf9f42caac3f6b63f8e317f329f2705320f02a330b243e1b605c6bc9b.json` (~76991 tok) - `2cbca2b1bf0345399a27e19131777f4d537a479e5d9eea2aefe60bfa6f0c848d.json` (~5279 tok) - `2d59df43c2cc4d09ea8566c4a94d70cf1a59b8d86e8fc4f9d7c49e92fd356d75.json` (~702 tok) - `2e097eddc32d49647dfd7b1c33bb454265b5587ec5bf148029c1ea496321073c.json` (~5588 tok) @@ -574,6 +581,7 @@ - `35f823871a12ed1809e20d70a663c8a8929fb35f3748705499e32df6cc4ec5c6.json` (~277 tok) - `35fd2a3ceb004b1d47fe5ccc509be97f4d380faebcfff32399237e9dfab59c00.json` (~6584 tok) - `367e004bbee4bd3a32e05eaf817f2dae69e39266c36b1649e4d3929362e383da.json` (~930 tok) +- `36fa47b81d63c37de1102693c833e5a1c113dcb414837b3643ba68ff94058545.json` (~592 tok) - `3702a2af17c73eed0470df46cecddc0b4c9a93aaab893ff5292d5c5b9b4ed512.json` (~9636 tok) - `3714706b1961d4945195e8613ba7a5fcbf550c040e671d5860f2e8d747dc477c.json` (~6708 tok) - `37a543436eff088729eaa2a790c9ef9e58f59f226122e12aa2a0e6c38bc6f5a0.json` (~1870 tok) @@ -596,6 +604,7 @@ - `3d0a807870ec7f6703eab25022ba70e6d12cca273ef820ea6e3bc8a31c97afa5.json` (~10333 tok) - `3d7e3a102cd4fc3bc27f599364f8f63ec75e6cb1ea3c47bcacc0e1261b485a15.json` (~3783 tok) - `3da9ee357a09d89a26df59bd0393606349c2d0849078091078924f60b5cdfd51.json` (~495 tok) +- `3db60e6e67d5bb6410285138cb4c838b8adb1d211ebe6ee0c4938820fad4c66b.json` (~269 tok) - `3db8027a5cdc79633520de23a59b83fb76ecfba3132ba8a4f52e785220157e70.json` (~336 tok) - `3ddf0434ea5d953052cfdb1597b14704a3692537f86a9fc442ce21024160eb1a.json` (~30420 tok) - `3de5f25efab468345f714829408944025550d1056ba2b46fa3f3f29bcd5a9573.json` (~2258 tok) @@ -607,7 +616,9 @@ - `3f8d83e0da73654fdad2d36d37d926e0fcf87f3fc645834c75938afed280aa20.json` (~81 tok) - `3f9cade10a0070913a41048bc2462ee7a4680399174c61c30c8d6e8a6e722cee.json` (~4780 tok) - `40138b45926e46347669dbe289bb2ea7597aedc0eb558bd0d39aacfef5df62e1.json` (~2333 tok) +- `406549c33ed009f33fce2953bac23493fac1a3b8ddbb7f4ededf56d4d4467d29.json` (~10135 tok) - `40a5802d6063d7dceabcd49010eac965e03377546e69b2839ce06e3bc19fb99b.json` (~3592 tok) +- `40a70514720bf07cfbe92de67954839e91019b1170af380b1570f710e739a49d.json` (~652 tok) - `40a727896bbebc2f6e00afb878b8c666f7fdefa3f34fea7215de4ed042087b96.json` (~81 tok) - `4109287f426902eeb3615a1912430eac8ca65558cc4ec26c781ea237d5460ef0.json` (~28018 tok) - `41bc353adb8853621ebd82fa735c038f98fe3a7e14d44a2eba6a50f01b4cc944.json` (~1511 tok) @@ -625,6 +636,7 @@ - `45a3f01bceef855fff2282e1e2f5a96edaf6e49eb2dd6774de2c951190939476.json` (~876 tok) - `461865ad5d4eb698d9f8b0d0189052bd8733835f159fdec76ee474deb74fac89.json` (~466 tok) - `468a56306bfbb4b7f8f052404e6f4d239bde83c10b206520cb8e9fd6631647c4.json` (~1038 tok) +- `46caff82b95eaec8f55497010d7bd131f4a8651ee883a4d3ea1a121cfe515618.json` (~81 tok) - `473aad72b667e57806070866e41e43c658d0c08d9422b2bab8641b027e29d2db.json` (~53573 tok) - `4761c9bb00e54bf17b43a2ebb04ba21b531111c34e93a9dbc957756076e78bf8.json` (~25212 tok) - `480dbaa39314c68c3d4da3c1cb184846dc98ea3f672516afb1683ad674bc88cf.json` (~678 tok) @@ -658,6 +670,7 @@ - `51103bc49cc87487f9efa30dcffc5f7639f0cc6cbf8acc722f5da33fd0ce32e3.json` (~8308 tok) - `5175f1392d190fb958eb902f22aea50b3506648b739e0cb8a7bc42d389231e4b.json` (~4502 tok) - `51d646ce7e4c76a33d29d45d20149ff8825fe4339a842c2d000ed93d9e46f3de.json` (~12490 tok) +- `521cf1ac969a070d624e689274f75a76fc09d84e8a8ba86908b5a6178e776b03.json` (~10023 tok) - `527377e036844617174eac5b08965176bb127edc5db3d5a77f362b034a4ec4fd.json` (~10319 tok) - `5286722a3fefdea557784ae42c65304cf227e92c44178fff83b9f2394c57b2e6.json` (~5453 tok) - `528bfbd499082064cc9b015a4092ffba50f77e09f2f90b5497e20aec9aba24fb.json` (~138 tok) @@ -690,30 +703,12 @@ - `5a822398dceebb814bc41d8ffb801c51f89a129f9d090965877f7f67782a1b23.json` (~4032 tok) - `5a92249e0d4f9dd34ff89497b4c540e6b84a672a455b0a5be5ca710b94e9eb79.json` (~4454 tok) - `5a974e6340597c0447c72b0d7bc977a668a8da2fac8c90c810e886c319e9bbd0.json` (~2118 tok) -- `5a99edc88f1f75983b12b9b8931c7ba5717424024e2ffb000e19564f2956175c.json` (~2393 tok) -- `5b5887beb614d310bb85633443a1e29df0dec82a39650daecc39d529ffe5ce37.json` (~4806 tok) -- `5b710141d6eb0f199d33b695898e32519aca05ea857921dd62bf508c7f954948.json` (~567 tok) -- `5b791c1c235d96fe2e512c3d6fedd8f44c5ee256db33beb485935a3f6cc3d1c1.json` (~6930 tok) -- `5c1cd225c770bed4fc03e72d8433eb17a2193588c47d5abeb26580098aab6506.json` (~77 tok) -- `5cc2f13594e5c0d36031a44dc0b88706f3e547d5e48720b55f02a68eb3b3149b.json` (~789 tok) -- `5ce1db6b15faeeb658ef1fca18682532adca25278a2add781eb3c7caf4792b44.json` (~22459 tok) -- `5cebb6a86cbc5a90d803973f1365c1673f3a11bd36cfc5178bdc985a03950378.json` (~8230 tok) -- `5d7b2f244a9e7100f16139db782e80265d4724d5cb53b7c51fab1aeaaf1fb7b6.json` (~1313 tok) -- `5d996b732909e4428bb55522f4fdad491a98075c3aa256d7a816956bd42adac7.json` (~984 tok) -- `5da97b26e48c2ea5c6df990bedce598cfd8052022102fa7a7e4c499c6f1374f5.json` (~1773 tok) -- `5e6d0ac84e99bdd6f6129fa521f73726ded1a47ff2e5d46a7c69026cec7522a5.json` (~591 tok) -- `5eb698f53d93049c137820a52db3d1a7c451be085cea9d956a06b44671034787.json` (~1504 tok) ## src/controller/ -- `meson.build` (~1109 tok) -- `persist.c` — define _GNU_SOURCE (~13786 tok) +- `meson.build` (~1130 tok) ## src/controller/testsuite/ -- `controller_test.c` — include (~1346 tok) -- `test_commit_barrier_rollout.c` — RED tests for ha-commit-barrier-rollout 1.1: wal_classify totality (verified table), unknown-type fail-fast via fork+SIGABRT, fact-hook RETAIN routing (~4884 tok) - -## src/include/ - -- `persist.h` — pragma once (~2129 tok) +- `controller_test.c` — include (~1368 tok) +- `test_commit_barrier_reply.c` — Task 3.1 RED integration tests: commit-first-then-reply per opcode (jobctrl/queuectrl/hostctrl/worker-join) via endpoint outq reply capture + pthread handler runner; pins the key_find rc==1 hook-skip gap (~8639 tok) diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 608a058..387f3f3 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -7046,6 +7046,174 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-06-12T00:51:52.673Z" + }, + { + "id": "bug-438", + "timestamp": "2026-06-12T03:52:57.142Z", + "error_message": "Incorrect value in code", + "file": "src/controller/meson.build", + "root_cause": "Had 'commit.c'", + "fix": "Changed to 'testsuite/test_commit_barrier_rollout.c'", + "tags": [ + "auto-detected", + "wrong-value", + "build" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T03:52:57.142Z" + }, + { + "id": "bug-439", + "timestamp": "2026-06-12T04:13:06.326Z", + "error_message": "Missing error handling in unknown", + "file": ".kiro/specs/ha-commit-barrier-rollout/tasks.md", + "root_cause": "Code path had no error handling — exceptions would propagate uncaught", + "fix": "Added try/catch block", + "tags": [ + "auto-detected", + "error-handling", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T04:13:06.326Z" + }, + { + "id": "bug-440", + "timestamp": "2026-06-12T04:18:19.158Z", + "error_message": "Missing guard clause", + "file": "src/controller/persist.c", + "root_cause": "No early return/throw for edge case: detail == HOOK_DETAIL_WORKER_JOIN ||\n detail == H", + "fix": "Added guard clause: if (detail == HOOK_DETAIL_WORKER_JOIN ||\n )", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T04:18:19.158Z" + }, + { + "id": "bug-441", + "timestamp": "2026-06-12T04:18:31.067Z", + "error_message": "Significant refactor of ", + "file": "src/controller/persist.c", + "root_cause": "4 lines replaced/restructured", + "fix": "Rewrote 7→12 lines (4 removed) | Also: return persist_wal_async(p, wal_type, &js, sizeof(; persist_t* p = cls; | Also: int rc = persist_wal_async(p, wal_type, pkt, (uint; persist_t* p = cls;", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 3, + "last_seen": "2026-06-12T04:19:00.652Z" + }, + { + "id": "bug-442", + "timestamp": "2026-06-12T04:50:32.154Z", + "error_message": "Incorrect value in code", + "file": "src/controller/testsuite/test_commit_barrier_rollout.c", + "root_cause": "Had \"xce/types.h\"", + "fix": "Changed to \"xce/errnox.h\"", + "tags": [ + "auto-detected", + "wrong-value", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T04:50:32.154Z" + }, + { + "id": "bug-443", + "timestamp": "2026-06-12T05:14:36.392Z", + "error_message": "Missing guard clause", + "file": "src/controller/persist.c", + "root_cause": "No early return/throw for edge case: p == NULL", + "fix": "Added guard clause: if (p == NULL)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T05:14:36.392Z" + }, + { + "id": "bug-444", + "timestamp": "2026-06-12T09:09:04.495Z", + "error_message": "Significant refactor of ", + "file": "src/controller/testsuite/test_commit_barrier_reply.c", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 3→5 lines (2 removed) | Also: static xce_signal_t g_sigkill = { .signo = SIGKIL; static xce_jobctrl_t g_jobctrl;", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-06-12T09:09:10.703Z" + }, + { + "id": "bug-445", + "timestamp": "2026-06-12T09:30:00Z", + "error_message": "AddressSanitizer SEGV in hash_string via makeResource when running controller_unit_test -t test_commit_barrier_reply standalone", + "file": "src/controller/testsuite/test_commit_barrier_reply.c", + "root_cause": "Global xstring table uninitialised in standalone runs: makeString() returns NULL, resobj name becomes NULL, key_add hashes NULL. In full-suite runs test_worker_join.c:520 initialises the table unguarded, so the bug only appears with -t selection.", + "fix": "Probe makeString(\"cbrp-probe\") in the group entry; call xstring_table_init(139) only when the probe returns NULL (avoids double init in full-suite order).", + "tags": [ + "testsuite", + "xstring", + "fixture", + "segv", + "ha-commit-barrier-rollout" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T09:30:00Z" + }, + { + "id": "bug-446", + "timestamp": "2026-06-12T09:30:00Z", + "error_message": "makeJob() returns NULL (Expected Non-NULL) in commit-barrier reply fixture", + "file": "src/controller/testsuite/test_commit_barrier_reply.c", + "root_cause": "job_init (lib/jobj.c) returns -EINVAL when xce_job_t.submit == NULL, and makeJobsub requires job_obj_init() module tables which no controller test group initialises.", + "fix": "Fixture calls job_obj_init(64, 64) once per process (static guard) and provides a zeroed static job_submit_data_t as jdef.submit.", + "tags": [ + "testsuite", + "jobj", + "fixture", + "ha-commit-barrier-rollout" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T09:30:00Z" + }, + { + "id": "bug-447", + "timestamp": "2026-06-12T09:30:00Z", + "error_message": "queue_ctrl/worker_ctrl append NO WAL entry and reply status 1 on success; job_ctrl also returns 1 (key_find found-rc leak)", + "file": "src/controller/queue.c", + "root_cause": "key_find (xkvmgr_find, lib/kvmgr.c:130-151) returns 1 when the key is found. queue_ctrl (queue.c:60), worker_ctrl (worker.c:643) and job_ctrl (job.c:314) store it in rc and never reset it: the 'if (rc == 0)' AFTER_EVENT hook gate is skipped (no WAL_QUEUE_CTRL / WAL_WORKER_CTRL append on the found path) and the requester reply carries status 1 instead of 0. Production gap discovered by task-3.1 RED tests; queue/host control state changes are currently never persisted or replicated.", + "fix": "NOT fixed here (test-only task). Pinned RED by test_commit_barrier_reply.c append assertions; the task-3.2 handler restructure (rc = ctrl fn; if rc==0 barrier; reply rc) must normalise ctrl-fn rc semantics so the hook fires and success replies are 0.", + "tags": [ + "production-gap", + "queue_ctrl", + "worker_ctrl", + "job_ctrl", + "key_find", + "wal", + "ha-commit-barrier-rollout", + "red-finding" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-06-12T09:30:00Z" } ] } \ No newline at end of file diff --git a/.wolf/cerebrum.md b/.wolf/cerebrum.md index 7be787b..e6734f5 100644 --- a/.wolf/cerebrum.md +++ b/.wolf/cerebrum.md @@ -41,6 +41,9 @@ - Unity harness quirks in controller_unit_test (2026-06-11): (a) failure lines misattribute the FILE — Unity.TestFile comes from whichever group's UNITY_BEGIN ran, so a failure printed as test_repl_unit.c:294 may live in test_p2_integration.c; locate the actual test by grepping for the test NAME. (b) Each group's UNITY_BEGIN resets counters, so the process exit code only reflects the LAST group — failures in earlier groups are masked (exit=0); meson reports green despite real failures. Grep output for FAIL with `grep FAIL` on the raw log (ANSI codes sit between ':' and 'FAIL', so `grep ":FAIL"` misses them). - 3 pre-existing controller suite failures on persistent_ha baseline (2026-06-11, verified by stash/rebuild without ha-commit-engine changes, see bug-411): test_committed_lsn_does_not_advance_on_timeout (rc -110 vs expected 0 — persist_wal_sync_append now returns -ETIMEDOUT, test expects old rc==0+sync_rc contract), test_repl_ack_after_promote_gate_allows_jobsub and test_rdhup_primary_dispatch_gate_opens (both -10013: ha_has_write_quorum false, standby_endpoints empty with n_peers=2). - ha-commit-engine: task 1.2 shipped commit_on_membership_change as a FULL re-evaluation (commit_advance_locked + cond broadcast), and majority_candidate_locked already excludes in_catch_up + ep==NULL slots. Task 3.1 membership tests are green-on-arrival; task 3.2 is a no-op (verified 2026-06-11, tests 17-20 in test_commit_engine.c). +- key_find (xkvmgr_find) returns 1 when found, NOT 0 — queue_ctrl/worker_ctrl/job_ctrl leak this into rc: `if (rc == 0)` hook gates never fire on the found path (bug-447); any code reusing key_find's rc for success checks must normalise it. +- Controller test binary reply observation: no xce_reply hook exists in src/lib; observe replies on a zeroed no-iomgr endpoint_t — xce_reply endpoint_enq()s onto ep->outq; drain with endpoint_deq + raw capnp read_Request/read_RequestReply (test_commit_barrier_reply.c). +- makeJob needs job_obj_init() once per process AND xce_job_t.submit != NULL; makeString needs xstring_table_init (test_worker_join.c inits it unguarded at group entry — probe makeString for NULL before initialising to stay -t-selection safe). ## Do-Not-Repeat diff --git a/.wolf/memory.md b/.wolf/memory.md index 13888ab..ef9de7b 100644 --- a/.wolf/memory.md +++ b/.wolf/memory.md @@ -249,119 +249,13 @@ > Consolidated session (0 actions) ## Session: 2026-06-04 04:46 -> Consolidated session (144 actions) +> Consolidated session (0 actions) ## Session: 2026-06-05 18:53 - -| Time | Action | File(s) | Outcome | ~Tokens | -|------|--------|---------|---------|--------| +> Consolidated session (0 actions) ## Session: 2026-06-05 18:57 - -| Time | Action | File(s) | Outcome | ~Tokens | -|------|--------|---------|---------|--------| -| 19:12 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~18 | -| 19:19 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | modified XCE_TESTING() | ~2119 | -| 19:21 | Edited src/controller/ha.c | added 6 condition(s) | ~955 | -| 19:21 | Edited src/controller/ha.h | modified 1() | ~209 | -| 19:21 | Edited src/controller/ha.c | added 2 condition(s) | ~379 | -| 19:22 | Edited src/controller/ha.c | modified if() | ~27 | -| 19:41 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~23 | -| 19:46 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | modified XCE_TESTING() | ~1408 | -| 19:47 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 2→2 lines | ~41 | -| 20:05 | Edited src/controller/ha.c | modified if() | ~348 | -| 20:28 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~22 | -| 20:35 | Edited src/controller/config_repl.c | removed 207 lines | ~418 | -| 20:38 | Edited src/controller/testsuite/test_config_repl.c | modified test_ctrlhello_no_repl_messages_enqueued() | ~1189 | -| 20:38 | Edited src/controller/testsuite/test_config_repl.c | 3→5 lines | ~68 | -| 20:54 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~16 | -| 20:59 | Edited src/controller/testsuite/test_ha_role.c | modified XCE_TESTING() | ~1320 | -| 21:00 | Edited src/controller/testsuite/test_ha_role.c | 5→9 lines | ~135 | -| 21:01 | Edited src/controller/ha.c | added 7 condition(s) | ~846 | -| 21:02 | Edited src/controller/ha.h | modified node() | ~242 | -| 21:02 | Edited src/controller/ha.c | modified XCE_TESTING() | ~338 | -| 21:04 | Edited src/controller/ha.c | modified ha_check_majority_convergence() | ~574 | -| 21:42 | Edited src/controller/ha.h | 1→2 lines | ~34 | -| 21:42 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~15 | -| 21:49 | Edited src/controller/testsuite/test_ha_role.c | added 1 condition(s) | ~1918 | -| 21:52 | Edited src/controller/ha.h | 10000 → 60000 | ~14 | -| 21:52 | Edited src/controller/ha.h | 10000 → 60000 | ~30 | -| 21:53 | Edited src/controller/ha.c | added 1 condition(s) | ~441 | -| 21:53 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | inline fix | ~18 | -| 22:03 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~17 | -| 22:10 | Edited src/controller/config_repl.c | 6→11 lines | ~112 | -| 22:10 | Edited src/controller/config_repl.c | 11→16 lines | ~142 | -| 22:10 | Edited src/controller/config_repl.c | added 1 condition(s) | ~91 | -| 22:26 | Edited src/controller/ha.c | added 2 condition(s) | ~248 | -| 22:36 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~12 | -| 22:39 | Edited src/controller/testsuite/test_ha_role.c | modified test_evaluate_repl_action_stream_wal() | ~447 | -| 22:39 | Edited src/controller/testsuite/test_ha_role.c | 6→8 lines | ~127 | -| 22:39 | Edited src/controller/testsuite/test_config_repl.c | added error handling | ~1326 | -| 22:40 | Edited src/controller/testsuite/test_config_repl.c | 5→8 lines | ~123 | -| 22:59 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~17 | -| 22:59 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~15 | -| 23:02 | Edited src/controller/ha.h | 6→11 lines | ~213 | -| 23:03 | Edited src/controller/ha.c | my_lsn() → self_lsn() | ~247 | -| 23:03 | Edited src/controller/ha.c | modified if() | ~242 | -| 23:03 | Edited src/controller/ha.c | modified self_tag() | ~130 | -| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 18→20 lines | ~224 | -| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 9→11 lines | ~112 | -| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | modified XCE_TESTING() | ~418 | -| 23:04 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 5→7 lines | ~74 | -| 23:05 | Edited src/controller/testsuite/test_ha_evaluate_committing.c | 6→9 lines | ~108 | -| 23:05 | Edited src/controller/ha.h | fields() → fix() | ~179 | -| 23:06 | Edited src/controller/testsuite/test_config_repl.c | modified test_snapshot_recv_128kb_two_chunks() | ~1174 | -| 23:07 | Edited src/controller/testsuite/test_config_repl.c | 6→8 lines | ~123 | -| 23:20 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~16 | -| 23:20 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~13 | -| 23:21 | Session end: 54 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 22 reads | ~171678 tok | -| 23:41 | Edited src/controller/ha.c | 2→6 lines | ~118 | -| 23:42 | Edited src/controller/ha.c | 21→23 lines | ~362 | -| 06:50 | task 6.5 integration test: found bug in ha->n_peers (total=3 instead of other=2), fixed, retested — all checks pass | src/controller/ha.c | PASS majority convergence | ~8000 | -| 00:03 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~18 | -| 00:26 | Edited src/controller/ha.c | modified if() | ~144 | -| 00:31 | Edited src/controller/ha.c | modified liveness_poll_fn() | ~133 | -| 00:32 | Edited src/controller/ha.c | modified if() | ~170 | -| 00:36 | Edited src/controller/ha.c | modified XCE_TESTING() | ~165 | -| 01:02 | Edited src/controller/ha.c | 3→2 lines | ~38 | -| 01:06 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~18 | -| 01:26 | Edited src/controller/ha.c | added 4 condition(s) | ~1172 | -| 01:27 | Edited src/controller/ha.c | 13→8 lines | ~164 | -| 01:33 | Session end: 65 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~360703 tok | -| 01:37 | Edited src/controller/ha.c | count() → size() | ~102 | -| 01:37 | Edited src/controller/ha.c | 2→2 lines | ~31 | -| 01:37 | Edited src/controller/ha.c | modified for() | ~124 | -| 01:37 | Edited src/controller/ha.c | modified for() | ~162 | -| 01:38 | Edited src/controller/ha.c | modified for() | ~131 | -| 01:38 | Edited src/controller/ha.h | inline fix | ~29 | -| 01:38 | Edited src/controller/ha.h | modified node() | ~34 | -| 01:39 | Edited src/controller/testsuite/test_ha_role.c | n_peers() → immediately() | ~54 | -| 01:40 | Edited src/controller/testsuite/test_ha_role.c | 2 → 3 | ~17 | -| 01:40 | Edited src/controller/testsuite/test_ha_role.c | 10→10 lines | ~143 | -| 01:41 | Edited src/controller/testsuite/test_ha_role.c | 2 → 3 | ~15 | -| 01:42 | Edited src/controller/testsuite/test_ha_role.c | 7→7 lines | ~71 | -| 01:42 | Edited src/controller/testsuite/test_ha_role.c | 2 → 3 | ~19 | -| 01:42 | Edited src/controller/testsuite/test_ha_role.c | inline fix | ~13 | -| 01:47 | Session end: 79 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~361671 tok | -| 09:55 | E2E cluster lifecycle test (task 6.7) | logs/ctrl-*.log | PASS: build green, RDHUP→election→promotion, config tag preserved, ASAN clean | ~4k | -| 02:56 | Edited .kiro/specs/ha-repl-hardening/tasks.md | inline fix | ~12 | -| 02:57 | Session end: 80 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~361683 tok | -| 03:51 | Session end: 80 writes across 7 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 34 reads | ~361683 tok | -| 04:00 | Edited src/controller/ha.c | 8→5 lines | ~49 | -| 04:06 | Edited src/controller/ha.c | 6→8 lines | ~131 | -| 04:06 | Edited src/controller/ha.c | DEBUG2() → INFO() | ~36 | -| 04:28 | Edited src/controller/ha.c | modified if() | ~180 | -| 04:40 | Edited src/controller/ha.c | modified 1() | ~44 | -| 04:41 | Edited src/controller/ha.c | modified no() | ~121 | -| 04:46 | Edited src/controller/ha.c | INFO() → DEBUG2() | ~115 | -| 05:04 | Edited src/controller/ha.c | added 1 condition(s) | ~304 | -| 05:11 | Edited src/controller/ha.c | 2→1 lines | ~20 | -| 05:27 | Edited src/controller/ha.c | added 1 condition(s) | ~72 | -| 05:43 | Edited src/controller/ha.c | modified no() | ~52 | -| 06:02 | Edited src/lib/delay.c | added 5 condition(s) | ~270 | -| 06:18 | Edited src/controller/ha.c | reduced (-6 lines) | ~32 | -| 06:18 | Edited src/controller/ha.c | modified no() | ~52 | -| 06:28 | Session end: 94 writes across 8 files (tasks.md, test_ha_evaluate_committing.c, ha.c, ha.h, config_repl.c) | 35 reads | ~364937 tok | +> Consolidated session (102 actions) ## Session: 2026-06-08 01:03 @@ -1003,3 +897,31 @@ | 05:45 | Adversarial review ha-commit-barrier-rollout task 2.2 (barrier helper + walAck retirement): boundary clean (persist.c+repl.c only), 15/15 group, full census 472 Unity PASS / 0 FAIL (= 465 prior + 7 new), configAck byte-equivalent, LSN-0 absorption commented, ce==NULL asymmetry judged defensible — APPROVED | src/controller/persist.c, src/controller/repl.c | APPROVED | ~30k | | 22:36 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~21 | | 22:36 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | modified RESOLVED() | ~148 | + +## Session: 2026-06-12 00:02 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| + +## Session: 2026-06-12 00:39 + +| Time | Action | File(s) | Outcome | ~Tokens | +|------|--------|---------|---------|--------| +| 01:55 | Created src/controller/testsuite/test_commit_barrier_reply.c | — | ~7971 | +| 01:55 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified cbrp_op_thread_fn() | ~102 | +| 01:56 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 6→5 lines | ~87 | +| 01:56 | Edited src/controller/meson.build | 2→6 lines | ~37 | +| 01:56 | Edited src/controller/testsuite/controller_test.c | 1→2 lines | ~25 | +| 01:56 | Edited src/controller/testsuite/controller_test.c | 3→4 lines | ~27 | +| 01:57 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 2→3 lines | ~15 | +| 01:58 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified notify() | ~392 | +| 02:01 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~115 | +| 02:06 | Edited src/controller/testsuite/test_commit_barrier_reply.c | expanded (+6 lines) | ~132 | +| 02:08 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~200 | +| 02:09 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 3→5 lines | ~89 | +| 02:09 | Edited src/controller/testsuite/test_commit_barrier_reply.c | added 1 condition(s) | ~217 | +| 02:20 | Edited src/controller/testsuite/test_commit_barrier_reply.c | modified notify() | ~218 | +| 02:20 | Edited src/controller/testsuite/test_commit_barrier_reply.c | 46→42 lines | ~673 | +| 09:35 | Task 3.1 ha-commit-barrier-rollout: created RED integration tests for commit-first-then-reply (4 opcodes x withheld/acked/validation) | src/controller/testsuite/test_commit_barrier_reply.c, meson.build, controller_test.c | 484 results, 8 RED fails (new file only), 472 baseline green; discovered key_find rc==1 leak in queue_ctrl/worker_ctrl/job_ctrl (bug-447) | ~90k | +| 03:00 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~21 | +| 03:00 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | 1→3 lines | ~255 | diff --git a/src/controller/meson.build b/src/controller/meson.build index 8c6d922..5a9cdf4 100644 --- a/src/controller/meson.build +++ b/src/controller/meson.build @@ -122,7 +122,11 @@ controller_unit_test_src = [ 'testsuite/test_ha_liveness.c', 'testsuite/test_commit_engine.c', 'testsuite/test_commit_barrier_rollout.c', + 'testsuite/test_commit_barrier_reply.c', 'commit.c', + 'job.c', + 'queue.c', + 'join.c', 'startup_validation.c', 'worker.c', 'persist.c', diff --git a/src/controller/testsuite/controller_test.c b/src/controller/testsuite/controller_test.c index 64df06f..a41e6ed 100644 --- a/src/controller/testsuite/controller_test.c +++ b/src/controller/testsuite/controller_test.c @@ -52,6 +52,7 @@ extern void test_ha_rdhup_integration(void); extern void test_ha_liveness(void); extern void test_commit_engine(void); extern void test_commit_barrier_rollout(void); +extern void test_commit_barrier_reply(void); void setUp(void) { return; } @@ -173,6 +174,7 @@ int main(int argc, char* argv[]) TEST(test_ha_liveness); TEST(test_commit_engine); TEST(test_commit_barrier_rollout); + TEST(test_commit_barrier_reply); return UNITY_END(); } diff --git a/src/controller/testsuite/test_commit_barrier_reply.c b/src/controller/testsuite/test_commit_barrier_reply.c new file mode 100644 index 0000000..dac880c --- /dev/null +++ b/src/controller/testsuite/test_commit_barrier_reply.c @@ -0,0 +1,800 @@ +/* + * test_commit_barrier_reply.c — RED-phase integration tests for + * commit-first-then-reply per opcode (ha-commit-barrier-rollout task 3.1). + * Requirements 1.1 (success reply only after majority commit), 1.2 (commit + * timeout → -ETIMEDOUT error reply), 1.3 (barrier rejection → error reply, + * never success), 1.4 (no success-interpretable reply pre-commit). + * + * Drives the four request-shaped handlers DIRECTLY on a worker pthread + * (vtask-context stand-in; production dispatch: cluster.c:325-363) against + * a real tmpdir persist_t embedded in a test cluster_t with registered + * persistence hooks, a wired commit engine (n_peers=3), a stub standby + * registry (2 dummy standbys), ACKs driven through repl_on_ack: + * do_jobcontrol_op — not-started-job branch → WAL_JOB_FAIL (SIGKILL) + * do_queuecontrol_op — QUEUE_CLOSE → WAL_QUEUE_CTRL + * do_hostcontrol_op — HOST_CLOSE → WAL_WORKER_CTRL/UPDATE + * handle_worker_join — first-time join → WAL_WORKER_CTRL/JOIN + * (via handle_join — handle_worker_join is static) + * + * Reply observation harness: there is no xce_reply test hook in src/lib + * (out of boundary), so replies are observed where the production send + * path deposits them — xce_reply() encodes the reply Request and + * endpoint_enq()s it onto ep->outq (request.c:334-368, netio.c:796). The + * requester endpoint is a zeroed static endpoint_t with no iomgr: + * endpoint_update_epoll() is a no-op and the zeroed pthread mutexes are + * valid locks on Linux, so packets stay queued. Tests drain via + * endpoint_deq() and parse with the raw capnp reader (capn_init_mem → + * read_Request → read_RequestReply), the raw-packet parse pattern of + * test_config_repl.c — least invasive, zero production changes. + * + * RED expectations against the current reply-before-commit handlers + * (anchors re-verified 2026-06-12: job.c:932, queue.c:130, worker.c:700, + * join.c:107-109): + * - *_withheld_acks_replies_error / *_success_reply_only_after_commit: + * the handlers reply immediately, before any covering ACK → the + * Req 1.1/1.2/1.4 ordering assertions fail. Additional RED finding: + * queue_ctrl/worker_ctrl/job_ctrl leak key_find's found rc (== 1), so + * queue/host control skip their AFTER_EVENT hook (NO WAL entry) and + * reply status 1 — pinned by the append assertion, 3.2 must close it. + * - *_error_reply_immediate: local validation failures already reply + * immediately with their existing codes — contract pins that stay + * green before AND after the task-3.2 restructure. + * + * Fixtures reused from test_commit_barrier_rollout.c (engine + stub + * registry + drop send hook + LSN-0 priming append) and test_worker_join.c + * (kvmgrs, resources, hostattrs, hostconfig; its collect_confvars stub + * also serves this file's join path). + */ + +/* rtti.h must come first — it defines 'var' and 'typeinfo_t' used by kvmgr.h */ +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "controller.capnp.h" +#include "xce/cluster_def.h" +#include "xce/types.h" + +#include "cluster.h" +#include "clusterconfig.h" +#include "commit.h" +#include "hook.h" +#include "jobj.h" +#include "kvmgr.h" +#include "myconf.h" +#include "netio.h" +#include "persist.h" +#include "qobj.h" +#include "repl.h" +#include "resobj.h" +#include "request.h" +#include "typemap.h" +#include "wal_types.h" +#include "wobj.h" +#include "xstring.h" +#include "zarray.h" + +/* ── Link stubs ────────────────────────────────────────────────────────── + * job.c / join.c reference these cross-module functions (opupd.c, sched.c, + * apiserver.c — not part of the unit-test link). None of them sits on the + * four code paths under test (jobctrl not-started branch, queue/host ctrl, + * worker join); stubbed so the binary links, same precedent as + * test_worker_join.c's collect_confvars stub. */ + +/* clang-format off */ +int notify(cluster_t *c, endpoint_t *e, xce_notify_data_t *d) +{ (void)c; (void)e; (void)d; return 0; } +worker_t *get_ep_worker(cluster_t *c, endpoint_t *e) +{ (void)c; (void)e; return NULL; } +int update_host_runusage(xce_job_runusage_t **r, char *h, int n, + xce_rusage_t **u) +{ (void)r; (void)h; (void)n; (void)u; return 0; } +int update_pids(job_pidinfo_t **i, int n, int *p) +{ (void)i; (void)n; (void)p; return 0; } +int update_pgids(job_pidinfo_t **i, int n, int *p) +{ (void)i; (void)n; (void)p; return 0; } +int on_scheduler_join(cluster_t *c) { (void)c; return 0; } +int add_apiserver(cluster_t *c, char *n, endpoint_t *e) +{ (void)c; (void)n; (void)e; return 0; } +int on_apiserver_join(cluster_t *c, endpoint_t *e) +{ (void)c; (void)e; return 0; } +/* clang-format on */ + +/* Test-injectable send hook (repl.c, XCE_TESTING) — keeps repl_broadcast + * from dereferencing the dummy standby endpoints. */ +extern int (*repl_broadcast_send_hook)(void *ep, + wal_catchup_entry_t **entries, + int n_entries); + +static int cbrp_drop_send_hook(void *ep, wal_catchup_entry_t **entries, + int n_entries) +{ + (void)ep; (void)entries; (void)n_entries; + return 0; /* pretend success, drop the data */ +} + +static uint64_t cbrp_mono_ms(void) +{ + struct timespec ts; + clock_gettime(CLOCK_MONOTONIC, &ts); + return (uint64_t)ts.tv_sec * 1000u + (uint64_t)ts.tv_nsec / 1000000u; +} + +static void cbrp_sleep_ms(uint32_t ms) +{ + struct timespec ts; + ts.tv_sec = ms / 1000u; + ts.tv_nsec = (long)(ms % 1000u) * 1000000L; + nanosleep(&ts, NULL); +} + +/* ── Reply observation (see file header for the harness rationale) ───── */ + +typedef struct { + int n_replies; /* replymsg packets seen by this scan */ + int32_t last_status; /* status of the last replymsg observed */ + uint64_t committed_at_obs; /* committed_lsn sampled at observation time */ +} cbrp_reply_obs_t; + +static persist_t *g_obs_persist; /* sampled into committed_at_obs */ + +/* Drain every packet queued on ep->outq; record replymsg status. Non-reply + * packets (e.g. on_worker_join's config push) are parsed and discarded. */ +static void cbrp_scan_replies(endpoint_t *ep, cbrp_reply_obs_t *obs) +{ + data_item_t *it; + + while ((it = endpoint_deq(ep)) != NULL) { + io_packet_t *pkt = ITPTR_TO_PACKET(it); + struct capn c; + struct Request rq; + Request_ptr rp; + + capn_init_mem(&c, (const uint8_t *)pkt->data, pkt->sz, 0); + rp.p = capn_getp(capn_root(&c), 0, 1); + read_Request(&rq, rp); + if (rq.body_which == Request_body_replymsg) { + struct RequestReply rr; + read_RequestReply(&rr, rq.body.replymsg); + obs->n_replies += 1; + obs->last_status = rr.status; + obs->committed_at_obs = atomic_load(&g_obs_persist->committed_lsn); + } + capn_free(&c); + free_packet(pkt); + free(pkt); + } +} + +/* Poll until at least one replymsg is observed or the deadline expires. */ +static void cbrp_poll_reply(endpoint_t *ep, uint32_t deadline_ms, + cbrp_reply_obs_t *obs) +{ + uint64_t t0 = cbrp_mono_ms(); + for (;;) { + cbrp_scan_replies(ep, obs); + if (obs->n_replies > 0 || cbrp_mono_ms() - t0 >= deadline_ms) + return; + cbrp_sleep_ms(5); + } +} + +/* Poll until the WAL grows past lsn_before (the operation's append became + * visible) or the deadline expires; returns the appended watermark + * (current_lsn - 1) or 0 on timeout. */ +static uint64_t cbrp_poll_append(persist_t *p, uint64_t lsn_before, + uint32_t deadline_ms) +{ + uint64_t t0 = cbrp_mono_ms(); + for (;;) { + uint64_t cur; + pthread_mutex_lock(&p->lock); + cur = p->current_lsn; + pthread_mutex_unlock(&p->lock); + if (cur > lsn_before) + return cur - 1; + if (cbrp_mono_ms() - t0 >= deadline_ms) + return 0; + cbrp_sleep_ms(5); + } +} + +/* ── Handler-on-a-pthread runner (vtask-context stand-in) ───────────── */ + +typedef int (*cbrp_op_fn_t)(cluster_t *, endpoint_t *, xce_request_t *); + +typedef struct { + cbrp_op_fn_t fn; + xce_request_t *req; + int rc; +} cbrp_op_arg_t; + +/* Statics survive Unity's longjmp on assertion failure. */ +static cbrp_op_arg_t g_op_arg; +static pthread_t g_op_tid; + +static cluster_t g_hc; /* harness cluster (defined below) */ +static endpoint_t g_req_ep; /* requester endpoint (capture, defined below) */ + +static void *cbrp_op_thread_fn(void *arg) +{ + cbrp_op_arg_t *a = arg; + a->rc = a->fn(&g_hc, &g_req_ep, a->req); + return NULL; +} + +/* ── Cluster harness ─────────────────────────────────────────────────── */ + +/* Resource names/types required by init_worker → apply_join_state + * (add_hostattrs_resvals, worker.c) — mirror of test_worker_join.c. */ +#define CBRP_N_RES 9 +static const char *cbrp_res_names[CBRP_N_RES] = { + "type", "model", "cpuf", "maxcpus", "maxmem", + "maxswap", "maxtmp", "ndisks", "slots" +}; +static const int cbrp_res_valtypes[CBRP_N_RES] = { + XCE_VAL_STRING, XCE_VAL_STRING, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, + XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, XCE_VAL_NUMERIC, + XCE_VAL_NUMERIC +}; + +/* Statics survive Unity's longjmp on assertion failure. + * (g_hc / g_req_ep are tentatively defined above for the thread runner.) */ +static commit_engine_t g_hce; /* wired commit engine */ +static int g_hep_a, g_hep_b;/* dummy standby endpoints */ +static ep_private_t g_req_priv; +static xce_peerinfo_t g_req_from; +static char g_hc_tmpdir[256]; + +/* myconf scaffolding for find_matched_hostconfig (worker join path) */ +static myconf_t g_hc_myconf; +static cluster_config_t g_hc_clusterconf; +static xce_hostconfig_t g_hc_hconfig; +static xce_hostconfig_t *g_hc_hconfigs[1]; + +static void cbrp_setup(uint32_t sync_timeout_ms) +{ + persist_config_t cfg; + kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING, + .nslots = 64, .delta = 16 }; + kvmgr_params_t int_params = { .key_type = KVMGR_KEY_INT, + .nslots = 64, .delta = 16 }; + int i; + + memset(&g_hc, 0, sizeof(g_hc)); + memset(&g_req_ep, 0, sizeof(g_req_ep)); + memset(&g_req_priv, 0, sizeof(g_req_priv)); + memset(&g_req_from, 0, sizeof(g_req_from)); + + /* Requester endpoint: zeroed, no iomgr — xce_reply packets stay on + * ep->outq (see file header). */ + g_req_ep.fd = -1; + g_req_ep.data = &g_req_priv; + g_req_from.name = (char *)"requester"; + g_req_from.key = (char *)""; + + g_hc.myhost = &g_hc.myhost_info; + g_hc.myhost_info.name = (char *)"ctrl-test"; + g_hc.myhost_info.key = (char *)""; + + /* ha zero-init: CTRL_PRIMARY == 0 — handle_join's standby gates pass */ + TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&g_hc.hooksys, + XCE_CONTROLLER_HOOK_MAX)); + + g_hc.jobmgr = makeInstance(&xkvmgr_type, &int_params); + g_hc.qmgr = makeInstance(&xkvmgr_type, &str_params); + g_hc.workermgr = makeInstance(&xkvmgr_type, &str_params); + g_hc.resmgr = makeInstance(&xkvmgr_type, &str_params); + TEST_ASSERT_NOT_NULL(g_hc.jobmgr); + TEST_ASSERT_NOT_NULL(g_hc.qmgr); + TEST_ASSERT_NOT_NULL(g_hc.workermgr); + TEST_ASSERT_NOT_NULL(g_hc.resmgr); + zarray_init(&g_hc.sharedinst, 64, NULL); + + for (i = 0; i < CBRP_N_RES; i++) { + xce_resource_t rdef; + memset(&rdef, 0, sizeof(rdef)); + rdef.name = (char *)cbrp_res_names[i]; + rdef.valtype = cbrp_res_valtypes[i]; + rdef.idx = -1; + TEST_ASSERT_NOT_NULL_MESSAGE(makeResource(g_hc.resmgr, &rdef), + cbrp_res_names[i]); + } + + /* hostconfig "default" so find_matched_hostconfig succeeds for joins */ + memset(&g_hc_myconf, 0, sizeof(g_hc_myconf)); + memset(&g_hc_clusterconf, 0, sizeof(g_hc_clusterconf)); + memset(&g_hc_hconfig, 0, sizeof(g_hc_hconfig)); + g_hc_hconfig.name = (char *)"default"; + g_hc_hconfig.hosttype = (char *)"!"; + g_hc_hconfig.hostmodel = (char *)"!"; + g_hc_hconfig.mxj = 2; + g_hc_hconfigs[0] = &g_hc_hconfig; + g_hc_clusterconf.nhconfigs = 1; + g_hc_clusterconf.hconfigs = g_hc_hconfigs; + g_hc_myconf.cluster_config = &g_hc_clusterconf; + g_hc.myconf = &g_hc_myconf; + + /* Real persist over a private tmpdir; cluster back-pointer set by + * persist_init so persist_start registers the production hooks. */ + cfg = persist_config_defaults(); + cfg.wal_sync_ms = 5000; + cfg.sync_timeout_ms = sync_timeout_ms; + + snprintf(g_hc_tmpdir, sizeof(g_hc_tmpdir), "/tmp/xce_cbrp31_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(g_hc_tmpdir)); + persist_set_workdir(&g_hc.persist, g_hc_tmpdir, "ctrl"); + TEST_ASSERT_EQUAL_INT(0, persist_init(&g_hc.persist, &cfg, &g_hc)); + TEST_ASSERT_EQUAL_INT(0, persist_start(&g_hc.persist)); + + /* 3-member cluster: 2 stub standbys, engine wired, ACKs withheld + * until a test explicitly delivers them via repl_on_ack. */ + TEST_ASSERT_TRUE( + repl_add_standby(&g_hc.persist.repl, &g_hep_a, "sb_a", 7400) >= 0); + TEST_ASSERT_TRUE( + repl_add_standby(&g_hc.persist.repl, &g_hep_b, "sb_b", 7401) >= 0); + TEST_ASSERT_EQUAL_INT(0, commit_engine_init(&g_hce, &g_hc.persist, 3, 8)); + repl_set_commit_engine(&g_hc.persist.repl, &g_hce); + repl_broadcast_send_hook = cbrp_drop_send_hook; + + g_obs_persist = &g_hc.persist; + + /* Priming append so operation entries land at LSN >= 1 (the engine + * rejects lsn == 0 — accepted interim LSN-0 edge, ha-commit-engine). */ + TEST_ASSERT_EQUAL_INT( + 0, persist_wal_async(&g_hc.persist, WAL_JOB_UPDATE, "p", 1)); + TEST_ASSERT_EQUAL_UINT64(1, g_hc.persist.current_lsn); +} + +static void cbrp_teardown(void) +{ + cbrp_reply_obs_t drain = { 0 }; + + cbrp_scan_replies(&g_req_ep, &drain); /* free any leftover packets */ + repl_broadcast_send_hook = NULL; + commit_engine_fini(&g_hce); + persist_stop(&g_hc.persist); + persist_fini(&g_hc.persist); + + zarray_fini(&g_hc.sharedinst); + if (g_hc.resmgr) dereference(g_hc.resmgr); + if (g_hc.workermgr) dereference(g_hc.workermgr); + if (g_hc.qmgr) dereference(g_hc.qmgr); + if (g_hc.jobmgr) dereference(g_hc.jobmgr); + xce_hook_fini(&g_hc.hooksys); +} + +/* ── Per-opcode entity + request builders ────────────────────────────── */ + +/* job 42, submitted but never started (allocs == NULL): SIGKILL takes the + * not-started branch of job_ctrl → mark_job_fail → WAL_JOB_FAIL. */ +static xce_signal_t g_sigkill = { .signo = SIGKILL, .force = 0 }; +static xce_jobctrl_t g_jobctrl; +static xce_request_t g_jobctrl_req; +static job_submit_data_t g_job_submit; /* job_init requires submit != NULL */ + +static void cbrp_arrange_jobctrl(uint64_t jobid, int create_job) +{ + if (create_job) { + xce_job_t jdef; + static int jobobj_ready; /* jobj module tables — once per process */ + if (!jobobj_ready) { + TEST_ASSERT_EQUAL_INT(0, job_obj_init(64, 64)); + jobobj_ready = 1; + } + memset(&g_job_submit, 0, sizeof(g_job_submit)); + memset(&jdef, 0, sizeof(jdef)); + jdef.jobid = jobid; + jdef.state = JOB_STAT_PEND; + jdef.submit = &g_job_submit; + TEST_ASSERT_NOT_NULL(makeJob(g_hc.jobmgr, &jdef)); + } + memset(&g_jobctrl, 0, sizeof(g_jobctrl)); + g_jobctrl.action = JobControl_u_siginfo; + g_jobctrl.jobid = jobid; + g_jobctrl.u.siginfo = &g_sigkill; + memset(&g_jobctrl_req, 0, sizeof(g_jobctrl_req)); + g_jobctrl_req.opcode = OPCode_jobcontrol; + g_jobctrl_req.seq = 11; + g_jobctrl_req.from = &g_req_from; + g_jobctrl_req.body_which = Request_body_jobctrlmsg; + g_jobctrl_req.body.jobctrlmsg = &g_jobctrl; +} + +/* queue "q1": QUEUE_CLOSE → status |= QUEUE_STAT_CLOSED → WAL_QUEUE_CTRL */ +static xce_queuectrl_t g_queuectrl; +static xce_request_t g_queuectrl_req; + +static void cbrp_arrange_queuectrl(const char *qname, int create_queue) +{ + if (create_queue) { + xce_queue_t qdef; + memset(&qdef, 0, sizeof(qdef)); + qdef.queue = (char *)qname; + TEST_ASSERT_NOT_NULL(makeQueue(g_hc.qmgr, &qdef)); + } + memset(&g_queuectrl, 0, sizeof(g_queuectrl)); + g_queuectrl.action = QUEUE_CLOSE; + g_queuectrl.queue = (char *)qname; + memset(&g_queuectrl_req, 0, sizeof(g_queuectrl_req)); + g_queuectrl_req.opcode = OPCode_queuecontrol; + g_queuectrl_req.seq = 12; + g_queuectrl_req.from = &g_req_from; + g_queuectrl_req.body_which = Request_body_queuectrlmsg; + g_queuectrl_req.body.queuectrlmsg = &g_queuectrl; +} + +/* worker "hostA": HOST_CLOSE → status |= XCE_WORKER_CLOSED → + * WAL_WORKER_CTRL detail WORKER_UPDATE */ +static xce_hostctrl_t g_hostctrl; +static xce_request_t g_hostctrl_req; + +static void cbrp_arrange_hostctrl(const char *host, int create_worker) +{ + if (create_worker) { + xce_worker_t wdef; + hostattrs_t *attrs = calloc(1, sizeof(*attrs)); + TEST_ASSERT_NOT_NULL(attrs); + attrs->hosttype = makeString("LINUX"); + attrs->hostmodel = makeString("x86"); + attrs->cpufactor = 1.0f; + attrs->maxcpus = 4; + memset(&wdef, 0, sizeof(wdef)); + wdef.name = (char *)host; + wdef.attrs = attrs; + TEST_ASSERT_NOT_NULL(makeWorker(g_hc.workermgr, &wdef)); + } + memset(&g_hostctrl, 0, sizeof(g_hostctrl)); + g_hostctrl.action = HOST_CLOSE; + g_hostctrl.hostname = (char *)host; + memset(&g_hostctrl_req, 0, sizeof(g_hostctrl_req)); + g_hostctrl_req.opcode = OPCode_hostcontrol; + g_hostctrl_req.seq = 13; + g_hostctrl_req.from = &g_req_from; + g_hostctrl_req.body_which = Request_body_hostctrlmsg; + g_hostctrl_req.body.hostctrlmsg = &g_hostctrl; +} + +/* worker join: handle_join with role HostRole_worker reaches the static + * handle_worker_join; first-time join → init_worker → AFTER_EVENT hook → + * WAL_WORKER_CTRL detail WORKER_JOIN. */ +static xce_joindata_t g_joindata; +static hostattrs_t g_join_attrs; +static xce_peerinfo_t g_join_from; +static xce_request_t g_join_req; + +static void cbrp_arrange_worker_join(const char *host) +{ + memset(&g_join_attrs, 0, sizeof(g_join_attrs)); + g_join_attrs.hosttype = makeString("LINUX"); + g_join_attrs.hostmodel = makeString("x86"); + g_join_attrs.cpufactor = 1.0f; + g_join_attrs.maxcpus = 4; + g_join_attrs.maxmem = 16; + g_join_attrs.maxswap = 32; + g_join_attrs.maxtmp = 64; + g_join_attrs.ndisks = 1; + memset(&g_joindata, 0, sizeof(g_joindata)); + g_joindata.role = HostRole_worker; + g_joindata.data_which = JoinData_data_attrs; + g_joindata.data.attrs = &g_join_attrs; + memset(&g_join_from, 0, sizeof(g_join_from)); + g_join_from.name = (char *)host; + g_join_from.key = (char *)""; + memset(&g_join_req, 0, sizeof(g_join_req)); + g_join_req.opcode = OPCode_join; + g_join_req.seq = 14; + g_join_req.from = &g_join_from; + g_join_req.body_which = Request_body_joinmsg; + g_join_req.body.joinmsg = &g_joindata; +} + +/* ── Shared case drivers ─────────────────────────────────────────────── */ + +/* Withheld-ACK case (Req 1.2, 1.4): the requester reply must carry + * -ETIMEDOUT and no success reply may be observable while the operation's + * entry exceeds committed_lsn. + * RED: the current handlers reply SUCCESS immediately. */ +static void cbrp_drive_withheld(cbrp_op_fn_t fn, xce_request_t *req, + uint32_t sync_timeout_ms) +{ + cbrp_reply_obs_t obs = { 0 }; + uint64_t lsn_before, watermark, committed_final; + + pthread_mutex_lock(&g_hc.persist.lock); + lsn_before = g_hc.persist.current_lsn; + pthread_mutex_unlock(&g_hc.persist.lock); + /* Act: handler on a worker pthread (post-3.2 it blocks at the barrier) */ + g_op_arg.fn = fn; + g_op_arg.req = req; + g_op_arg.rc = -9999; + TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL, + cbrp_op_thread_fn, &g_op_arg)); + + /* The reply may legitimately take up to the commit timeout */ + cbrp_poll_reply(&g_req_ep, sync_timeout_ms + 1500, &obs); + pthread_join(g_op_tid, NULL); + + watermark = cbrp_poll_append(&g_hc.persist, lsn_before, 1000); + committed_final = atomic_load(&g_hc.persist.committed_lsn); + + /* Assert: the operation must produce its WAL entry — nothing to + * commit-confirm otherwise (Req 1.1). RED finding: queue_ctrl and + * worker_ctrl leak key_find's found rc (== 1) into the `if (rc == 0)` + * hook gate → NO WAL entry, status-1 reply (header; 3.2 closes it). */ + TEST_ASSERT_TRUE_MESSAGE(watermark > 0, + "request-shaped control op must append its WAL entry (queue_ctrl/" + "worker_ctrl currently skip the AFTER_EVENT hook: rc==1 leak)"); + /* ...which never committed (ACKs withheld) */ + TEST_ASSERT_TRUE_MESSAGE(committed_final < watermark, + "committed_lsn must stay below the entry with ACKs withheld"); + + /* ...the requester got exactly an error reply */ + TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0, + "the requester must receive a reply"); + TEST_ASSERT_TRUE_MESSAGE( + !(obs.last_status == 0 && obs.committed_at_obs < watermark), + "no success reply may be observable while the operation's entry " + "exceeds committed_lsn (Req 1.4)"); + TEST_ASSERT_EQUAL_INT_MESSAGE(-ETIMEDOUT, obs.last_status, + "withheld ACKs past the commit timeout must surface -ETIMEDOUT " + "to the requester (Req 1.2)"); +} + +/* Covering-ACK case (Req 1.1, 1.4): the success reply must be observable + * only after committed_lsn covers the operation's entry. + * RED: the current handlers reply success before any ACK arrives. */ +static void cbrp_drive_acked(cbrp_op_fn_t fn, xce_request_t *req) +{ + cbrp_reply_obs_t early = { 0 }; + cbrp_reply_obs_t final = { 0 }; + uint64_t lsn_before, watermark; + + pthread_mutex_lock(&g_hc.persist.lock); + lsn_before = g_hc.persist.current_lsn; + pthread_mutex_unlock(&g_hc.persist.lock); + g_op_arg.fn = fn; + g_op_arg.req = req; + g_op_arg.rc = -9999; + TEST_ASSERT_EQUAL_INT(0, pthread_create(&g_op_tid, NULL, + cbrp_op_thread_fn, &g_op_arg)); + + /* Wait until the operation's append is visible, then hold the handler + * blocked for a beat and sample the reply capture BEFORE any ACK. */ + watermark = cbrp_poll_append(&g_hc.persist, lsn_before, 2000); + cbrp_sleep_ms(150); + cbrp_scan_replies(&g_req_ep, &early); + + /* Deliver the covering ACK, then collect the (now permitted) reply */ + repl_on_ack(&g_hc.persist.repl, &g_hep_a, watermark); + cbrp_poll_reply(&g_req_ep, 2000, &final); + pthread_join(g_op_tid, NULL); + + /* Assert: the operation must produce its WAL entry (see the rc==1 + * leak note in cbrp_drive_withheld — same RED finding applies) */ + TEST_ASSERT_TRUE_MESSAGE(watermark > 0, + "request-shaped control op must append its WAL entry (queue_ctrl/" + "worker_ctrl currently skip the AFTER_EVENT hook: rc==1 leak)"); + + /* THE ordering gate: nothing the requester can read as confirmation + * may exist before the covering ACK (Req 1.1, 1.4) */ + TEST_ASSERT_EQUAL_INT_MESSAGE(0, early.n_replies, + "no reply may be observable before committed_lsn covers the " + "operation's entry (Req 1.1, 1.4)"); + + /* After the ACK: success reply, entry committed */ + TEST_ASSERT_TRUE_MESSAGE(final.n_replies > 0, + "the requester must receive the reply once the entry commits"); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, final.last_status, + "the post-commit reply must report success (Req 1.1)"); + TEST_ASSERT_TRUE_MESSAGE(final.committed_at_obs >= watermark, + "the success reply must be observable only with committed_lsn " + "covering the operation's entry (Req 1.1)"); +} + +/* Local-validation-failure pin (Req 1.4 boundary, task brief): the + * existing error reply stays immediate — no barrier wait, no WAL entry. + * Green before AND after the task-3.2 restructure (latency-bounded). */ +static void cbrp_drive_validation_error(cbrp_op_fn_t fn, xce_request_t *req, + int32_t expect_status, + int check_exact_status) +{ + cbrp_reply_obs_t obs = { 0 }; + uint64_t lsn_before, lsn_after, t0, elapsed_ms; + + pthread_mutex_lock(&g_hc.persist.lock); + lsn_before = g_hc.persist.current_lsn; + pthread_mutex_unlock(&g_hc.persist.lock); + /* Act: synchronous — a validation failure must never block */ + t0 = cbrp_mono_ms(); + (void)fn(&g_hc, &g_req_ep, req); + elapsed_ms = cbrp_mono_ms() - t0; + + cbrp_scan_replies(&g_req_ep, &obs); + pthread_mutex_lock(&g_hc.persist.lock); + lsn_after = g_hc.persist.current_lsn; + pthread_mutex_unlock(&g_hc.persist.lock); + + /* Assert: immediate error reply with the existing code, no append */ + TEST_ASSERT_TRUE_MESSAGE(elapsed_ms < 1000, + "local validation failures must reply without a barrier wait"); + TEST_ASSERT_TRUE_MESSAGE(obs.n_replies > 0, + "validation failure must still produce a requester reply"); + if (check_exact_status) + TEST_ASSERT_EQUAL_INT_MESSAGE(expect_status, obs.last_status, + "validation failure must keep its existing error code"); + else + TEST_ASSERT_TRUE_MESSAGE(obs.last_status < 0, + "validation failure must reply with an error status"); + TEST_ASSERT_EQUAL_UINT64_MESSAGE(lsn_before, lsn_after, + "a rejected operation must append no WAL entry"); +} + +/* ── do_jobcontrol_op (not-started branch → WAL_JOB_FAIL) ────────────── */ + +static void test_jobctrl_withheld_acks_replies_error(void) +{ + cbrp_setup(300); + cbrp_arrange_jobctrl(42, 1); + cbrp_drive_withheld(do_jobcontrol_op, &g_jobctrl_req, 300); + cbrp_teardown(); + printf("test_jobctrl_withheld_acks_replies_error: PASS\n"); +} + +static void test_jobctrl_success_reply_only_after_commit(void) +{ + cbrp_setup(1500); + cbrp_arrange_jobctrl(43, 1); + cbrp_drive_acked(do_jobcontrol_op, &g_jobctrl_req); + cbrp_teardown(); + printf("test_jobctrl_success_reply_only_after_commit: PASS\n"); +} + +static void test_jobctrl_unknown_job_error_reply_immediate(void) +{ + cbrp_setup(3000); + cbrp_arrange_jobctrl(9999, 0); /* job does not exist */ + cbrp_drive_validation_error(do_jobcontrol_op, &g_jobctrl_req, + -ENOENT, 1); + cbrp_teardown(); + printf("test_jobctrl_unknown_job_error_reply_immediate: PASS\n"); +} + +/* ── do_queuecontrol_op (QUEUE_CLOSE → WAL_QUEUE_CTRL) ───────────────── */ + +static void test_queuectrl_withheld_acks_replies_error(void) +{ + cbrp_setup(300); + cbrp_arrange_queuectrl("q1", 1); + cbrp_drive_withheld(do_queuecontrol_op, &g_queuectrl_req, 300); + cbrp_teardown(); + printf("test_queuectrl_withheld_acks_replies_error: PASS\n"); +} + +static void test_queuectrl_success_reply_only_after_commit(void) +{ + cbrp_setup(1500); + cbrp_arrange_queuectrl("q2", 1); + cbrp_drive_acked(do_queuecontrol_op, &g_queuectrl_req); + cbrp_teardown(); + printf("test_queuectrl_success_reply_only_after_commit: PASS\n"); +} + +static void test_queuectrl_unknown_queue_error_reply_immediate(void) +{ + cbrp_setup(3000); + cbrp_arrange_queuectrl("no-such-queue", 0); + cbrp_drive_validation_error(do_queuecontrol_op, &g_queuectrl_req, + -ENOENT, 1); + cbrp_teardown(); + printf("test_queuectrl_unknown_queue_error_reply_immediate: PASS\n"); +} + +/* ── do_hostcontrol_op (HOST_CLOSE → WAL_WORKER_CTRL/UPDATE) ─────────── */ + +static void test_hostctrl_withheld_acks_replies_error(void) +{ + cbrp_setup(300); + cbrp_arrange_hostctrl("hostA", 1); + cbrp_drive_withheld(do_hostcontrol_op, &g_hostctrl_req, 300); + cbrp_teardown(); + printf("test_hostctrl_withheld_acks_replies_error: PASS\n"); +} + +static void test_hostctrl_success_reply_only_after_commit(void) +{ + cbrp_setup(1500); + cbrp_arrange_hostctrl("hostB", 1); + cbrp_drive_acked(do_hostcontrol_op, &g_hostctrl_req); + cbrp_teardown(); + printf("test_hostctrl_success_reply_only_after_commit: PASS\n"); +} + +static void test_hostctrl_unknown_host_error_reply_immediate(void) +{ + cbrp_setup(3000); + cbrp_arrange_hostctrl("no-such-host", 0); + cbrp_drive_validation_error(do_hostcontrol_op, &g_hostctrl_req, + -ENOENT, 1); + cbrp_teardown(); + printf("test_hostctrl_unknown_host_error_reply_immediate: PASS\n"); +} + +/* ── handle_worker_join via handle_join (→ WAL_WORKER_CTRL/JOIN) ─────── */ + +static void test_worker_join_withheld_acks_replies_error(void) +{ + cbrp_setup(300); + cbrp_arrange_worker_join("workerA"); + cbrp_drive_withheld(handle_join, &g_join_req, 300); + cbrp_teardown(); + printf("test_worker_join_withheld_acks_replies_error: PASS\n"); +} + +static void test_worker_join_success_reply_only_after_commit(void) +{ + cbrp_setup(1500); + cbrp_arrange_worker_join("workerB"); + cbrp_drive_acked(handle_join, &g_join_req); + cbrp_teardown(); + printf("test_worker_join_success_reply_only_after_commit: PASS\n"); +} + +static void test_worker_join_no_hostconfig_error_reply_immediate(void) +{ + cbrp_setup(3000); + cbrp_arrange_worker_join("workerC"); + + /* Remove every hostconfig: find_matched_hostconfig returns NULL and + * handle_worker_join must reply its existing error immediately — + * the (future) barrier sits between the AFTER_EVENT hook and the + * SUCCESS reply only; error replies stay immediate. */ + g_hc_clusterconf.nhconfigs = 0; + g_hc_clusterconf.hconfigs = NULL; + cbrp_drive_validation_error(handle_join, &g_join_req, 0, 0); + cbrp_teardown(); + printf("test_worker_join_no_hostconfig_error_reply_immediate: PASS\n"); +} + +/* ── Test group entry point ──────────────────────────────────────────── */ + +void test_commit_barrier_reply(void) +{ + /* makeString needs the global xstring table. In the full suite, + * test_worker_join already initialised it (unguarded, its line 520); + * standalone (-t test_commit_barrier_reply) it is uninitialised and + * makeString returns NULL. Probe instead of double-initialising. */ + if (makeString((char *)"cbrp-probe") == NULL) + xstring_table_init(139); + + UNITY_BEGIN(); + /* do_jobcontrol_op — Req 1.1, 1.2, 1.4 */ + RUN_TEST(test_jobctrl_withheld_acks_replies_error); + RUN_TEST(test_jobctrl_success_reply_only_after_commit); + RUN_TEST(test_jobctrl_unknown_job_error_reply_immediate); + /* do_queuecontrol_op */ + RUN_TEST(test_queuectrl_withheld_acks_replies_error); + RUN_TEST(test_queuectrl_success_reply_only_after_commit); + RUN_TEST(test_queuectrl_unknown_queue_error_reply_immediate); + /* do_hostcontrol_op */ + RUN_TEST(test_hostctrl_withheld_acks_replies_error); + RUN_TEST(test_hostctrl_success_reply_only_after_commit); + RUN_TEST(test_hostctrl_unknown_host_error_reply_immediate); + /* handle_worker_join (via handle_join) */ + RUN_TEST(test_worker_join_withheld_acks_replies_error); + RUN_TEST(test_worker_join_success_reply_only_after_commit); + RUN_TEST(test_worker_join_no_hostconfig_error_reply_immediate); + UNITY_END(); +}