fix(xce-test-ha): chaotic mid-round walcompare false committed divergence (bug-1853)

A chaotic round's walcheck compared all three members' on-disk WALs with
boundary=auto. The just-killed term-4 primary still held its NACKed,
never-committed lsn=58, which differed from the new term-5 leader's
own-term noop at lsn=58, and was counted as class=committed.

walcheck_count now takes an optional "midloop" mode, used only by the
per-round chaotic check: the boundary is the serving primary's own
committedLsn (live_committed_boundary() needs 3-of-3 convergence, which a
chaotic round never reaches), and only members guaranteed to hold the
committed prefix are compared -- the primary plus live standbys whose
appliedLsn has reached the boundary. Every other caller, including the
settled final sweep, still compares all three. Log lines name members=.

Also logs bug-1854 (scheduler re-adds every configured resource on each
rejoin config push; ERROR noise, changed definitions ignored) for
follow-up.

10-round chaotic soak: PASS (divergence 0, job loss 0, SIMVERIFY 9 files,
0 mismatches).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.wolf/buglog.json

@@ -25204,6 +25204,36 @@
         "MOVE_PTR",
         "driver"
       ]
+    },
+    {
+      "id": "bug-1853",
+      "timestamp": "2026-09-24T08:20:00Z",
+      "error_message": "Test-harness false positive, not a controller safety bug. 10-round chaotic soak (run-ha-soak.sh --chaotic 10): VERDICT committed_divergence=1 committed_job_loss=0 result=FAIL. Single hit, round 3, mid-loop walcompare with boundary=auto: 'WALCMP_DIVERGENCE lsn=58 class=committed compared=3 present=3'. Final settled sweep: committed prefix consistent, all 33 jobs present.",
+      "file": ".claude/skills/xce-test-ha/run-ha-soak.sh",
+      "root_cause": "LSN 58 was never committed. ctrl-3 (term-4 primary) appended lsn=58 at 08:06:22, its only standby NACKed it ('WAL NACK from standby lsn=58 success=0'), acks never passed acked_lsn=57; the soak then killed ctrl-3. New primary ctrl-1 (term 5) correctly appended its own-term noop at lsn=58 (08:06:35). The round-3 walcheck fell back to boundary=auto (live_committed_boundary unavailable right after the election) and auto-derive (min CRC-intact tip across ALL members' on-disk WALs, including the just-killed ex-primary's) put lsn 58 inside the 'committed' window, so the dead ex-primary's uncommitted term-4 tail vs the new term-5 entry was classified as class=committed. ctrl-3 truncated/overwrote it on rejoin; settled sweep consistent. Same family as the transient-artifact false positives documented at live_committed_boundary()/bug-1487.",
+      "fix": "FIXED (2026-09-24, harness only). walcheck_count() takes an optional 2nd arg 'midloop', passed only by run_chaotic_rounds' per-round check: it takes the serving primary's own committedLsn (probe_primary, up to 3 tries; live_committed_boundary() needs 3-of-3 convergence, which a chaotic round never reaches -- first cut retried it and always fell back to auto) before falling back to auto, and compares only live members guaranteed to hold the committed prefix -- the serving primary plus live standbys whose node_probe appliedLsn >= boundary (all live members under auto); with fewer than 2 such members the mid-loop check is logged as skipped. Every other caller, including the settled final sweep, keeps the unchanged full 3-member comparison, so an excluded member is still checked once it has caught up. Log lines now name members=. SKILL.md chaotic-verdict section updated. Candidate (c), re-check-after-settle, rejected: a real divergence can 'heal' onto wrong content. Validated: 10-round chaotic soak PASS (committed_divergence=0, job_loss=0).",
+      "tags": [
+        "ha-soak",
+        "chaotic",
+        "walcompare",
+        "false-positive",
+        "test-harness"
+      ]
+    },
+    {
+      "id": "bug-1854",
+      "timestamp": "2026-09-24T08:35:00Z",
+      "error_message": "OPEN, NOT FIXED, low priority. Every HA soak run logs 'ERROR [sched_init_resources] sched_init_resources add resource <name> failed' for EVERY configured resource (~33 lines), once per scheduler process, ~30s after it joined -- i.e. when a primary failover makes the scheduler re-join and the controller re-pushes its config (231/264/429 lines in 3 soak runs today). Scheduling is unaffected in those runs (all soaks PASS, replay parity 0 mismatches).",
+      "file": "src/scheduler/src/scheduler.c",
+      "root_cause": "sched_init_config() is written as a first-time initializer but runs on every config push (sched_apply_pending_config, including each rejoin's push). sched_init_resources() calls makeResource() for each config->resources[i], which fails when the name is already registered in core.resmgr (key_add of an existing key) and only logs. Consequences: (1) ERROR log noise on every rejoin; (2) latent: a rejoin/upgrade config push that CHANGES a resource definition (flags, valtype, order) never updates the scheduler's copy. Pre-existing -- the same call ran on the handler thread before bug-1846's deferral; not introduced by it. sched_init_queues() re-adding queues on each push is worth checking under the same lens.",
+      "fix": "NOT FIXED. Candidate: on an existing name, update the registered xce_resource_t in place (or treat identical definitions as a no-op and log at DEBUG), mirroring core_events.c's core_upsert_* upsert posture for RESOURCES_PUSH; decide separately whether a changed definition needs picker/accounting refresh.",
+      "tags": [
+        "scheduler",
+        "config-push",
+        "rejoin",
+        "resources",
+        "log-noise"
+      ]
     }
   ]
 }