configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
7 cases: two covered short-circuits, timeout with committed_lsn
unmoved, covering-ACK release, sync_timeout_ms fallback (bounded both
directions), n_peers=1 standalone, NOT_READY passthrough via cap
saturation. Watermark resolved to current_lsn - 1 (design's literal
phrasing unsatisfiable per persist.h:71 invariant) with off-by-one
killing power both ways. 15 tests / 7 RED; suite baseline zero.
cluster_t embeds commit_engine_t; init in cluster_init_phase2 after the
vtaskmgr guard (idempotence-guarded — phase2 can run twice), wiring
repl_set_commit_engine + ha.repl back-pointer before any standby can
register; fini after iomgr/vtask shutdown. persist_wal_sync_append
passes written_lsn; engine-wired path drops -ENODEV and the
committed_lsn store (engine owns advancement); LSN-0 first-entry
routes to legacy per the accepted interim. p2 -ENODEV pin updated to
the Req 6.3 contract. 34/34 + 6/6 green through honest delegation;
census exactly at the 3 validated baselines; production binary links.
replicator_t gains ce pointer (set-once-before-traffic contract).
repl_on_ack feeds commit_on_ack after releasing r->lock (iomgr-safe);
four membership mutators feed commit_on_membership_change post-unlock,
gated on actual mutation. walAck branch delegates to commit_submit(WAIT)
when ce wired AND entry_lsn != 0 (LSN-0 interim guard until 5.3);
legacy counting kept for ce==NULL, marked DEPRECATED. commit.c linked
into production target (inert until 5.3 wiring). Tests 30-31 + all
repl_majority walAck cases green; 33/34 remain RED for 5.3; baseline
unchanged.
Tests 21-28: repl_active_standby_count contract (5 RED vs inert -1
stub) + 3 gate-level pins (n_peers<=1 always-true, zero-connected
reject, stateless re-accept) that 4.2 must preserve. Declaration in
repl.h; ha.c/ha.h untouched (4.2 boundary). 28 tests / 5 RED; suite
at the 3-failure baseline with unchanged signatures.
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Add self-referential entries @3 :List(WalCatchupEntry) field with
$C.maplistcount("n_entries"). Single-entry path unaffected — n_entries
defaults to 0, entries to NULL when not using batch mode.
Add StateReport struct {appliedLsn, configTag} as stateReportMsg @15
variant in RequestReply.data union. Standby populates this in heartbeat
replies so the primary can track per-standby convergence.
capnpc-c 1.0.1 has a bug with inline :group inside codecgen unions;
workaround uses named outer struct (wire-equivalent). typemap.h
extended with state_report_t and union member for C codec access.
- config_repl.c: add handle_repl_config_push — standby handler for
OPCode_replConfigPush. Verifies SHA-256, saves blob to disk via
conf_save_config, reloads via load_primary_config, ACKs primary.
- config_repl.h: declare handle_repl_config_push.
- cluster.c: dispatch OPCode_replConfigPush to handle_repl_config_push;
populate c->myhost_info.name from myconf->name (ctrl-1/ctrl-2/ctrl-3)
instead of the static "controller" placeholder.
- ha.c: fix raft_cb_become_leader push_config_to_standbys call to pass
real config bytes and tag read from disk (was NULL/0 — standbys got
empty push and never saved config).
- typemap.h: add raw_sha256/raw_bytes/raw_bytes_len fields to
repl_config_push_req_t (mirrors upgrade_config_req_t pattern).
- request.c: pre-extract List(UInt8) sha256/bytes from OPCode_replConfigPush
before capn_free() in xce_decode(); free them in xce_free_request().
- helper.c: add all HA/replication opcode names to xce_request_op_name
so "unknown opcode 30" becomes "replConfigPush" in logs.
- cluster_def.h: add embedded myhost_info field to cluster_t struct.
- ha.h: declare ha_start_primary_heartbeat; remove dead
ha_find_and_connect_to_primary declaration.
Add void (*on_connected)(endpoint_t *ep) field to endpoint_t with
endpoint_set_on_connected() helper. io_handler fires it exactly once
immediately after ssl_handshake_done=1 then nulls the pointer.
Enables ha_start_election_async to send OP_requestVote as soon as
the TLS handshake with a peer completes — no blocking SSL_connect.
6 new unit tests pass; controller suite 1/1 OK.
Add millisecond-granularity delay_task_ms() to delaymgr using
clock_gettime(CLOCK_MONOTONIC) for deadline computation. Extends
delay_task_t with nextrun_ts field; delay_worker distinguishes
ms-tasks from second-tasks via non-zero nextrun_ts. Backward
compatible with existing delay_task callers.
4 delay tests pass (2 new: tolerance and invalid-args).
Provides the timeout primitive for election and committing watchdogs.
Add ctrl_tunables struct to cluster_t with per-table fields:
jobmgr_nhash/delta, workermgr_nhash/delta, qmgr_nhash/delta,
resmgr_nhash/delta/inst, xstrings_nhash
init_controller_tunables() sets defaults (jobs=512, others=128)
and is called from cluster_init_phase1 so values are available
before cluster_init_phase2. Phase2 reads from c->tunables instead
of cluster_config or local #defines.
Each table gets its own values since they hold different item counts:
jobs can scale to hundreds, workers/queues/resources are typically
single-digit to tens.
The vote_request_thread was sending raw capnp(RequestVoteRequest) bytes
but the server-side iomgr decodes every incoming packet as a Request
struct. This caused the term field (uint64) to be misread as the opcode
field (uint16), so vote RPCs were silently misrouted (term=1 → OPCode_reply,
term=2 → OPCode_hello) and no node ever got a vote from its peers.
Fix:
- controller.capnp: add requestvotemsg @27 to Request.body and
requestvotereplymsg @14 to RequestReply.data
- typemap.h: move request_vote_req_t/reply_t before xce_request_t;
add requestvotemsg/requestvotereplymsg to body/data unions
- ha.c: rewrite vote_request_thread send/recv to use xce_encode()
wrapper and decode outer Request→inner RequestReply on receive
- cluster.c: add case OPCode_requestVote handler calling
handle_request_vote() and replying via xce_reply()
All 188 unit tests pass.
- errnox.h: add E_XCE_NOT_READY and E_XCE_STANDBY error codes with
extern int XCE_ERR_NOT_READY / XCE_ERR_STANDBY runtime variables
- ha.h: declare ha_dispatch_gate(ha_state_t *, int opcode)
- ha.c: define XCE_ERR_* variables; initialize heartbeat_seen semaphore
and primary_ctrl_fd = -1 in ha_init(); sem_destroy in ha_fini();
implement ha_dispatch_gate() with allowlist for CTRL_INIT/CTRL_ELECTING
(only OP_requestVote and OP_probeConfigTag pass) and CTRL_STANDBY
(replication, election, and housekeeping ops pass; primary-only ops
return -XCE_ERR_STANDBY)
- cluster.c: wire ha_dispatch_gate() in myreq_handler() before opcode
switch; gate failure sends error reply and returns early
All 393 unit tests pass. Satisfies requirements 1.3 and 1.4.
- Add --vcpus N argp option in main.c; store in myconf.vcpus
- Replace inline argp guard with validate_vcpus() call (returns -EINVAL for <=0)
- Add validate_startup_config() in startup_validation.c: returns 0; writes
deprecation warning into caller-supplied warn_buf when vcpus key present
- Wire validate_startup_config() into myapp_run() after conffile resolved;
emit WARN() if warn_buf non-empty (Req 4.4)
- Remove vcpus field from cluster_config_t (clusterconfig.h)
- Preserve capnp ordinal: vcpus @7 -> vcpusDeprecated @7: Void in controller.capnp
- vtaskmgr_init uses only myconf->vcpus with hardcoded default 8
- Add stub C types in typemap.h for capnp schema extensions from task 1.1
- 96 tests pass
xce_replconfig_reply_raw() - primary sends reply with real bytes via endpoint
xce_replconfig_req_ssl() - standby sends request over raw SSL socket
xce_replconfig_reply_ssl_decode() - standby decodes reply from raw SSL
xce_upgrade_config_req_extract() - documents architectural limitation; returns -ENOTSUP
xce_upgrade_config_req_extract_raw() - extract bytes from raw packet for OP_upgradeConfig
Same bypass pattern as existing xce_replconfigpush_req_raw(): build the
capnp message directly via capn_new_list8 + capn_setv8, sidestepping the
encode_Data_ptr path that routes raw bytes through xce_data_t (a
discriminated cluster-data union that cannot carry raw byte arrays).
Unblocks handle_config_request() and do_one_config_fetch() wiring.
Task A research finding: xce_upgrade_config_req_extract(xce_request_t*)
is unimplementable because xce_decode() calls capn_free() before any
handler runs, and decode_Data_ptr converts the wire-format List(UInt8)
into a zero-filled xce_data_t. Task D must capture the raw io_packet_t
bytes before xce_decode tears them down, then call the _raw variant.
Crash: encode_NameValue_list called with count=21, s=NULL on worker
rejoin, causing SIGSEGV in persist_worker_hook_fun.
Root cause: free_name_val_list(xce_name_val_t***, int) accepted count by
value. It zeroed *vals (resvals_to_sched = NULL) but could not zero
nresvals_to_sched. Any caller that freed the scheduler resource snapshot
(opget.c workerlist, reply.c clusterconf, dataex.c, sched.c dispatch)
left nresvals_to_sched stale. When the worker later rejoined and the
WAL HOOK fired, the WAL encoder read count=21 with a NULL pointer.
Fix: change signature to free_name_val_list(xce_name_val_t***, int*)
so both fields are zeroed atomically. Update all 7 call sites.
When a worker re-joins and already exists in workermgr, makeWorker_()
now overrides the existing slot instead of returning EEXIST. Admin-set
flags (LOCKEDU, CLOSED, HIDDEN, CLOSED_BY_ADMIN) survive the override.
Add XCE_WORKER_CLOSED_BY_ADMIN flag to distinguish admin-closed hosts
from workload-closed ones. worker_ctrl sets/clears it alongside CLOSED.
Refactor worker_init/worker_destroy into copy_worker/kill_worker helpers
shared by both fresh creation and override paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wire up 6 previously-stubbed message handlers in the scheduler:
resource dataset, resval dataset, job update notify, queue update
notify, resval update notify, and job rusage notify. Remove the
unreachable pushmsg branch (vtask_entry rewrites all pushmsg to
datamsg before the handler fires).
Changes by component:
- helper.h/helper.c: add xce_notify_event_name() for symbolic log names
- scheduler.h: add 5 new xce_sched_op_t values and matching union fields
- plugin.h/plugin.c: add plugin_job_rusage_hook_t and plugin_job_rusage_update()
- schedobj.c: add sched_update_existing_queue() with NULL/-ENOENT guards
- scheduler.c: wire 5 new dequeue cases; add sched_update_existing_job()
and static sched_op_name(); emit WARN log on handler failure
- handler.c: implement sched_handle_resources/resvals (dataset) and
sched_handle_sched_update_{job,queue,resvals}/job_rusage (notify);
remove dead sched_handle_jobpush/workerpush and pushmsg branch;
update TRACE logs to use symbolic name helpers
- test.c: 5 new unit tests covering notify event names, enum values,
and queue update NULL guard (15/15 pass)
WAL payloads:
- Job state transitions (START/FINISH/FAIL/STOP/RESUME/UPDATE etc):
wal_jobstate_t {uint64_t jobid, uint32_t state} — restores exact state flags
- Queue state (QUEUE_CTRL): wal_queuehdr_t {uint32_t status} + name string —
re-applies admin open/close/active/inactive after config queues are built
- Worker CTRL: existing [uint32_t detail][capnp xce_worker_t] unchanged
New recovery functions:
- restore_job_state(c, jobid, state): looks up job, overwrites state flags
- restore_queue_state(c, name, status): looks up queue, overwrites status
apply_wal_entry: dispatches all WAL types including job state and queue
test_repl_apply.c: weak stubs for restore_job_state/restore_queue_state
so unit test binary links without job.c/queue.c
Three identical static copies (controller, worker, scheduler) replaced
with a single xce_parse_peers() in src/lib/cluster.c.
The libxce version is the canonical implementation: stores the logical
name from 'name=host:port' syntax so daemons on the same dev host can
identify themselves by name rather than IP address.
- xce_candidate_t gains name[] field for logical peer identity
- parse_peers stores name prefix before '=' (empty when omitted)
- cluster_init matches own identity by name first, hostname as fallback
enabling multiple controllers on the same dev host via
--peers ctrl-1=10.0.1.1:7000,ctrl-2=10.0.1.2:7001
- myconf_t gains peers_cfg (xce_cluster_config_t) populated from --peers
- controller/main.c: parse --peers host:port,... via argp; ca_cert_path
auto-derived from workdir/certs/xce.crt (no --ca-cert flag needed)
- cluster_init: derive my_priority from position in peers list (index 0 =
primary); fall back to 0 when --peers not given (single-node mode)
Implement the catch-up state machine as testable pure-logic functions:
- repl_start_catchup(): marks standby in_catch_up=1, stores join_applied_lsn
- repl_complete_catchup(): clears in_catch_up=0, broadcasts ack_cond so any
blocked repl_wait_ack() re-checks the ACK condition immediately
- repl_needs_snapshot(): pure predicate — returns 1 when standby LSN predates
oldest surviving WAL segment (snapshot transfer required), 0 otherwise
Also fixes pre-existing linker error: repl.c was missing from controller_src
(xce_controller production binary), causing undefined references to repl_*
symbols introduced in P2-4.
7 new TDD tests in test_repl_catchup.c — all pass. Existing tests green.
- Move repl.h from controller/ to include/ (shared header) to break
persist.h ↔ repl.h circular include
- Extract wal_entry_hdr_t + wal_entry_type into include/wal_types.h so
repl.h can include only what it needs without pulling in all of persist.h
- Add replicator_t repl field to persist_t (embedded by value, P2-4)
- Wire repl_init/repl_fini into persist_init/persist_fini lifecycle
- Wire repl_broadcast into persist_wal_async after each WAL write;
last_replicated_crc updated here (not in persist_write_checkpoint_marker)
- Wire repl_wait_ack into persist_wal_sync_append; returns 0 on local
WAL success; sync_rc_out carries -ENODEV/-ETIMEDOUT for degraded mode
- Add 4 TDD tests in test_persist_repl.c: structural repl field check,
async no-standbys succeeds, sync degraded mode rc, checkpoint CRC invariant
Upgrade apply_wal_entry() from P0 static no-op stub to the real public
standby replay path:
- Reject oversized payloads (> MAX_WAL_PAYLOAD_BYTES) with -EMSGSIZE
- Skip WAL_CHECKPOINT (100) entries silently — local-only markers, never
replicated to standbys; current_lsn is NOT advanced on skip
- Skip unknown types (> WAL_ENTRY_TYPE_MAX) with WARN log and rc=0 for
forward compatibility with WAL written by newer controller versions
- Advance current_lsn to hdr->lsn+1 and update last_crc under p->lock
for all accepted known entries
- Gate restore_job()/worker_from_capnp() calls on p->cluster != NULL so
unit tests work without a full cluster wired in
- Add NULL guard returning -EINVAL for missing p or hdr
Add public declaration in persist.h with forward-declare stubs for
restore_job/worker_from_capnp in persist.c (avoids heavy cluster.h chain).
Test suite: 5 new Unity tests in test_repl_apply.c covering all branches.
Weak stubs for restore_job/worker_from_capnp in the test file satisfy the
linker without linking job.c/worker.c's heavy dependency graph.
All 5 new tests pass; full controller suite green.
