Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
controller.capnp declared `struct Data $C.mapname("xce_data_t")` (an
OPCode_get query-result tagged union) at file scope, which shadows
Cap'n Proto's builtin Data blob type for every later `: Data` field in
the same file. This silently miscompiled 7 fields across 6 structs as
a struct-pointer type instead of a raw byte blob: UpgradeConfigRequest
.sha256/.bytes, ReplConfigReply.sha256/.bytes, ReplConfigPushRequest
.sha256/.bytes, ReplEnvelope.payload, SnapshotTransfer's chunk
payload, WalCatchupEntry.payload/.batchPayloads, and CtrlHelloRequest
.key. Production C code had already routed around it for years via
hand-rolled raw capn_ptr manipulation (see request.c's own comments);
nothing before now needed the generated codec/accessors to actually
work for these fields.
Renamed the local struct to XceData (capnp-level name only; the
$C.mapname("xce_data_t") C typedef is unchanged) and explicitly
repointed getdata/pushmsg at XceData by name, since those two fields
correctly want the local tagged union. Updated every derived
generated-identifier reference across 19 hand-written files spanning
controller, lib, scheduler, and the Go apiserver/xcecli CGo bindings.
Fixed 5 call sites in lib/request.c that had been typed against the
old workaround alias but now legitimately receive the correct
capn_data type (a pure type-name swap -- Data_ptr/XceData_ptr and
capn_data share the identical {capn_ptr p} layout).
Verified via full rebuild+test: lib (1/1), controller (3/3, incl. the
job-array push regression tests), scheduler (15/15), Go apiserver +
xcecli (all packages green), and a direct pycapnp round-trip proving
UpgradeConfigRequest.sha256/.bytes now encode/decode correctly.
Bug: bug-1461 (.wolf/buglog.json). Unblocks admin-config-tool task 3.3.
extract_data() (src/lib/request.c), the shared reassembly step every
inbound push passes through inside vtask_entry() before any receiver's
handler runs, had no case for Data_data_jobArray -- it hit
default: rc=-EINVAL. Because vtask_entry()'s callnext check treats any
nonzero return (including a negative errno) as "reassembly complete,
proceed," this never surfaced as an error: the real payload was freed
and the receiver silently handed an empty, zeroed dataset_t, with no
log anywhere in the chain. Tasks 5.1-5.3 were each individually
correct and tested in isolation, but the feature was dead on the real
wire end-to-end.
Adds case Data_data_jobArray: passing the single, unchunked
descriptor straight through (no pagination needed, unlike the
existing LIST-shaped cases) and nulls the source pointer after
transfer -- verified this is not cosmetic, since free_Request()'s
pushmsg case would otherwise double-free it via free_Data()'s live
Data_data_jobArray branch.
Proven with a real xce_encode()/xce_decode() round-trip through the
fixed extract_data() -> sched_handle_dataset() -> sched_on_job_array()
(task 5.3, unmodified) chain, not just a unit-level check on
extract_data() alone. Also confirms task 5.4's future API-server
receive path shares this exact fix, since both go through the same
xcecli_ctx_init() -> vtask_entry() -> extract_data() pipeline.
One self-flagged, non-blocking residual gap (pre-existing since task
5.3, out of this task's request.c-only boundary): dataset->rows for
the jobArray case is never deep-freed downstream, leaking one bounded
xce_job_submit_t per successfully-delivered array push commit.
Task 5.3b of job-array-single-wal-record (Requirements 6.1, 6.2, 6.4, 8.4).
Inserted mid-run after independent review confirmed the gap; user
confirmed sequencing (fix now, before task 5.4).
Found via extended chaotic /xce-test-ha soaking on top of the bug-1419 fix
(same session). Both bugs independently caused the 10-round chaotic soak to
fail; sequential and chaotic now pass cleanly and repeatably.
1. Term-desync liveness deadlock (new): a standby that loses a near-
simultaneous election race to a peer can legitimately end up at a HIGHER
term than the peer that actually won and became primary. The primary's
now-stale-term AppendEntries/heartbeats are correctly rejected by the raft
library, but every one of those rejected messages was ALSO resetting the
standby's own liveness deadline (ha_on_heartbeat_received), so its
liveness timeout never fired and it could never start a fresh,
higher-term election to let the cluster self-heal — a permanent stall.
Fix: ha_msg_term_is_stale() gates the liveness reset (and
ha_discover_primary step-down) on the incoming message's own term stamp,
skipping it for provably-stale messages, in cluster.c's wal_catchup_entry
and nop/heartbeat handlers.
2. bug-847 (root-caused 2026-06-30, never fixed): a per-sub-entry `payload`
Data blob in a multi-entry WAL catch-up batch loses its capnp pointer once
the message crosses a segment boundary — the receiver decodes
raw_payload_len=0 for entries at/after the boundary and silently persists
empty WAL entries (structurally intact: correct LSN/term/CRC chain, empty
content). Only surfaces when a node rejoins and catches up a real batch,
which is why it needed a live chaotic soak (not the 2-entry unit test) to
reproduce. Fix: carry every sub-entry's payload concatenated in ONE
top-level batchPayloads blob instead of one Data blob per entry — a
single top-level blob does not exhibit the cross-segment loss. Sliced
back apart on decode using each sub-entry's own wirePayloadLen, a plain
scalar that (like lsn/entryType/term) decodes via the generic generated
decode path and was never subject to the bug. Confirmed via targeted
diagnostic logging that the corrupted entry arrives already-empty over
the wire (not a receive-side race), and via a faithful reproduction at
live JOB_SUBMIT payload size (5×3560B) after the original 2-entry/
14×872B tests turned out not to cross a segment boundary in this
environment.
Incidental fixes found and fixed along the way:
- A real self-deadlock (persist_entry_term() re-acquiring p->lock from
inside a caller that already held it) introduced by an initial, ultimately
unnecessary WAL-read locking fix attempted before bug-847 was pinned down;
fixed via a new persist_entry_term_locked() lock-free counterpart. Left in
place as a genuine (if incidental) hardening of ha_stream_wal_to_standby's
read consistency.
- .claude/skills/xce-test-ha/SKILL.md's `context: fork` frontmatter caused
the skill's own forked sub-session teardown to SIGHUP the whole
xce_controller process group mid-soak, killing all 3 nodes silently with
no error logged, whenever the soak was launched via the Skill tool itself
(setsid/nohup inside run-ha-soak.sh only protects against a controlling
terminal's own SIGHUP, not a parent forked-session's teardown). Removed;
added explicit guidance to launch long soaks via direct Bash from the main
session instead.
Verified: full 78-group unit regression sweep clean (0 failures) after each
change; two consecutive clean 10-round chaotic soaks and one clean 10-round
sequential soak on the final binary (zero committed divergence, zero job
loss, all 3 nodes led in both modes).
Adds checkpointTerm (@6, append-only) to the SnapshotTransfer wire
message. The leader stamps it via persist_entry_term(p, checkpoint_lsn)
when sending a snapshot; the standby reconciles it into its own
per-LSN term index (persist_reconcile_checkpoint_term) right after
persist_load_snapshot succeeds, before applied_lsn advances. This
authoritatively overwrites any stale local term a divergent
pre-snapshot entry left at the checkpoint LSN, closing the gap task
3.1's RED test proved (Expected 7 Was 3 -> now passes). Transport
stays persist_t-free; the reconcile lives in persist.c per the
existing dependency direction. Decode is auto-generated via capnp
mapname; only the hand-rolled encode path (request.c/typemap.h, needed
for the raw snapshot payload) required a matching manual update.
test_ha_core_safety 25/0, test_config_repl 66/0, no ASAN reports.
Closes bug-990: the live replicator repl_broadcast wire-encoded term=0
(entry.term was never set), so standbys persisted hdr.term=0 and took the
legacy direct-apply path instead of the library (term,lsn) intake. Now the
live broadcast and catch-up streamer carry the leader's real term + AE prev
tuple; ha_apply_wal_catchup_entry routes every term-carrying entry through
raft_on_append_entries_request -> on_persist_entry (leader term stamped),
retiring the term-0 direct-apply. Standbys now persist the leader's real term
and reconcile by (term,lsn); the live commit chain is end-to-end.
848/849/850 + persist-before-ACK + commit-barrier + bug-960 preserved.
Proven at integration level; the definitive live STRICT_TERM failover run is
env-blocked here (bug-924 overlayfs) and recommended on a real fs.
Send the HA heartbeat as OP_nop + the existing HeartbeatMsg body carrying
{term, committed_lsn, committed_config_tag} (senderPriority unused). Generic
xce_nop_req untouched; C-side body support pre-existed (no typemap edit). Req 1.1
producer half. Round-trip test proves the 3 fields survive the wire; lib suite
0 failures under ASAN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add io_classify_io_error and propagate endpoint/epoll errors through
endpoint_update_epoll and cbuf_get instead of swallowing them, so client
failover surfaces the real error rather than hanging. Adds netio arm-failure,
close-propagation, and request send-propagation tests.
Pre-existing uncommitted work, committed to establish a clean baseline before
ha-commit-durability implementation.
cluster_start (which set c->running) is orphaned on the HA myapp_run path, so
the running flag was never set on HA nodes — check_controller() treated the
primary as shutting down and zeroed joblist/workerlist reply chunks. Own the
flag lifecycle in cluster_run, symmetric with the c->running=0 on shutdown.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
xce_replconfig_reply_raw() - primary sends reply with real bytes via endpoint
xce_replconfig_req_ssl() - standby sends request over raw SSL socket
xce_replconfig_reply_ssl_decode() - standby decodes reply from raw SSL
xce_upgrade_config_req_extract() - documents architectural limitation; returns -ENOTSUP
xce_upgrade_config_req_extract_raw() - extract bytes from raw packet for OP_upgradeConfig
Same bypass pattern as existing xce_replconfigpush_req_raw(): build the
capnp message directly via capn_new_list8 + capn_setv8, sidestepping the
encode_Data_ptr path that routes raw bytes through xce_data_t (a
discriminated cluster-data union that cannot carry raw byte arrays).
Unblocks handle_config_request() and do_one_config_fetch() wiring.
Task A research finding: xce_upgrade_config_req_extract(xce_request_t*)
is unimplementable because xce_decode() calls capn_free() before any
handler runs, and decode_Data_ptr converts the wire-format List(UInt8)
into a zero-filled xce_data_t. Task D must capture the raw io_packet_t
bytes before xce_decode tears them down, then call the _raw variant.
Add the full call chain for host control operations from the Go API
server through the C library to the controller:
- schema/controller.capnp: HostControl struct + hostctrlmsg @21
in Request.body union
- include/typemap.h: xce_hostctrl_t* hostctrlmsg in body union
(placed after @20 clrpreasonmsg, matching capnp ordinal)
- include/client.h, xceapi.h, request.h: declarations for
xcecli_hostctrl, xce_hostctrl, xce_hostctrl_req
- lib/client.c: xcecli_hostctrl (validate + delegate to xce_hostctrl)
- lib/xceapi.c: xce_hostctrl (validate ctx state + send + wait reply)
- lib/request.c: xce_hostctrl_req (validate hostname/empty/nil,
accept only HOST_OPEN/HOST_CLOSE, reject HOST_TAG/UNTAG)
- lib/helper.c: Request_body_hostctrlmsg cases in
xce_request_body() and xce_request_msg_name()
- controller/cluster.c: OPCode_hostcontrol dispatch
- controller/include/cluster.h: do_hostcontrol_op declaration
- controller/worker.c: do_hostcontrol_op + fix pre-existing
goto-clean bug in worker_ctrl that leaked write lock and
reference on HOST_OPEN/HOST_CLOSE no-op paths
TDD: 8 xce_hostctrl_req validation tests (null ep, null ctrl,
null/empty hostname, HOST_DO_NONE no-op, HOST_TAG/UNTAG reject,
negative action reject, HOST_OPEN/CLOSE accept via xce_request).
116/116 xcelib tests pass. CGO_ENABLED=0 Go backend tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
