Move xce_test's 15 production sources + headers from src/controller/testsuite/
into a new src/harness/ component with its own meson.build, and strip every
now-dangling controller-side reference (11 controller_unit_test_src entries,
4 standalone test-binary blocks, 11 controller_test.c extern/RUN_TEST lines)
so the controller build is never left broken.
Fixes a broken relative-symlink bridge (xce_test_controller_link) added to
keep two controller-embedded tests (test_test_cli.c, test_xcetest_process_entry.c)
finding the relocated xce_test binary until task 2.1 moves them too.
Discovered and documented in design.md: four controller-domain regression
tests call functions directly from four of xce_test's own files, requiring
controller_unit_test to permanently recompile them from src/harness/ — a
real dependency neither the original discovery nor design audit caught.
controller_unit_test: 1228 Tests 0 Failures 0 Ignored (verified via the
Unity binary's own printed summary line, not just meson's aggregate count).
Task 1.1 of xce-harness-component.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
wal_type_name() never had a case for WAL_CONFIG_MARKER (type 50) -- every
config-change marker entry printed as UNKNOWN instead of a real, defined
type. Added the case plus payload decode (tag/sha256) matching config_repl.c's
config_marker_encode/_decode byte layout, hand-decoded here rather than
linking config_repl.c into xce_test's link graph.
Gap detection also had no way to tell "committed prefix missing!" (real
loss) apart from a normal checkpoint-triggered WAL trim (persist.c's
trim_wal_segments, floor-guarded so it never removes a segment a standby
still needs) -- every long-running node's WAL, once checkpointed, would
false-alarm on every inspection. Cross-checks the sibling data/snap/
directory (snapshot LSN is literally in the filename, snap-<hex>.capnp)
and downgrades the message when a snapshot covers the gap.
Adds waldump_stats_t.leading_gap_explained and 3 new tests plus one
assertion on the existing gap test.
Extends the HA soak harness with a new --array-soak mode: small (N=2)
and large (N=10000) array submission sub-rounds, each with one clean
kill/failover, asserting one WAL record per submission regardless of
array size, per-element addressability (status/cancel/completion)
spread across low/mid/high indices post-failover, and a real
millisecond-resolution replay-time bound comparing large-array vs
small-array kill-to-addressable latency.
waldump.c gains a WAL_JOB_ARRAY_SUBMIT case in wal_type_name() so the
scenario can grep an exact type name for its live WAL-record-count
delta check.
Went through one remediation round: the replay-time measurement
initially used whole-second resolution (collapsed to 0s/n-a on a real
run) and never asserted on the comparison, giving zero actual evidence
for the O(1)-replay-time requirement. Fixed with $EPOCHREALTIME
(millisecond resolution -- this sandbox's BusyBox date silently
mangles the more common `date +%s%3N` idiom) and a real bound
assertion that hard-fails the run on a genuine O(N) regression,
independently verified to have real teeth.
Live-verified 3 times with genuinely discriminating numbers each time;
final run: small=210ms, large=163ms -- large-array replay was not
slower, live evidence of the feature's O(1) claim.
Task 6.1 of job-array-single-wal-record (Requirements 4.3, 5.2, 5.3,
8.2, 8.3). Depends on and reuses task 6.0a's own soak verification.
Adds --factstream HA soak mode driving real jstart/jfinish/jfail worker-fact
traffic against a live 3-node cluster, killing the primary mid-stream and
asserting zero committed-state divergence and zero phantom fact reactions.
Building this scenario surfaced bug-1416: persist_recover()'s boot-time
apply-correctness verification can race the live apply pump and the standby
live-replication apply path, both of which mutate the same last_applied_lsn/
job-table state the verification re-reads live -- producing a false
APPLY_DIVERGENT that permanently fences an otherwise-healthy node. Fixed with
a new persist_t.recovery_complete gate that defers only the in-memory apply
(never the durability write or replication ACK) while a recovery pass is in
flight, closed from the earliest safe point in myapp_run() so an election-
retry candidate can't apply live traffic before persist_recover() ever runs.
Round-2 adversarial review independently verified the ACK/durability paths
stay ungated, reproduced the myapp.c fix as load-bearing via revert-and-
reproduce, and confirmed real-divergence detection still fires. Live 3-node
factstream soak: 2/2 rounds PASS, zero APPLY_DIVERGENT across 5 recover
calls. controller_unit_test: zero new failures across 23 groups.
Diffs member WALs by (lsn -> term, pcrc) over the committed prefix, exits non-zero
on any committed-prefix divergence with the diverging LSN + per-member entry
identities. Reuses the canonical waldump decoder via a new per-entry visitor seam
(no second WAL parser). Detects same-LSN/diff-term and same-LSN/diff-content
divergences (the bug-851 class) that an LSN-range check would mask.
