A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.
- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
answered. A separate opcode because replies are matched through the
client's single wbuf, not by sequence number: answering plain notifies
would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
do_notify, then waits on the commit barrier and replies 0 only once it
is majority-committed. A barrier failure (the fact is still appended and
retained) replies -E_XCE_NOT_READY, so the worker retries in place
instead of rotating away from a healthy primary (cf. bug-1850). Admitted
by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
(a controller predating the opcode) they fall back to the plain
fire-and-forget notify, so workers and controllers can be upgraded in
either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
retry budget (the rejoin resync remains the backstop). These calls run
on the worker daemon's IPC callback, so while the controller lacks
quorum other jobs' IPC messages wait (buffered, not lost) up to the
retry budget.
Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The primary echoes its own last_known_elected_at_ms on every outgoing
heartbeat (threaded through xce_ha_heartbeat_req's existing scalar-arg
plumbing, alongside term/committed_lsn/committed_config_tag); a standby
processing a received heartbeat (ha_consume_heartbeat_committed, the sole
production call site) stores it unconditionally into its own
last_known_elected_at_ms, reconcile-free -- no other HA field moves.
ha_observe_heartbeat_committed itself is untouched; the new store sits
next to its call site, not inside it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RLSNJhNoULvVcUhXEFafQC
Add xce_job_state_query_req()/xce_job_state_report_req() to request.c,
paginating job IDs / job-state entries across OPCode_jobStateQuery /
OPCode_jobStateReport messages via direct Request.body union arms
(mirroring xce_jobdispatch_req/xce_notify_req's direct-send shape, not
xce_push_data_req's XceData-table multiplexing -- these are dedicated
union arms, not multiplexed push/reply-get traffic). 8 round-trip tests
(normal/empty/multi-page for both query and report, covering all 4
JobStateStatus values and both queried states) all pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y4M5MJM2zK6nU43da267Ku
Adds the spec's one and only src/controller/ change: a new
case OPCode_schedAdmin in myreq_handler() (cluster.c) that mirrors the
existing OPCode_upgradeConfig case -- primary-only gate, explicit reply
on every path, stateless pass-through with no WAL entry.
Forwards to the scheduler via a new xce_schedadmin_req() (async send,
mirrors xce_fetchfile_req) and blocks the vtask thread on a condvar with
a 5s deadline while holding c->sched.lock for the round-trip, matching
this spec's own research-phase design decision (parity with
OPCode_upgradeConfig's no-RBAC posture, blocking wait accepted for
low-frequency admin traffic). A new branch in the existing
OPCode_reply case demuxes the scheduler's SchedAdminReply to a
process-global single-slot pending-waiter mechanism, serialized by
c->sched.lock, matching job.c's existing fetchfile-proxy pattern.
Adversarial review independently traced double-free/cleanup, the
timeout/signal race, and reply-opcode demux correctness against source;
found and a follow-up round closed a narrow-window leak via a carefully
lock-ordered unlock->unregister->relock sequence (avoiding an AB-BA
inversion against schedadmin_signal()'s lock acquisition order).
New test_sched_admin_dispatch.c drives the real myreq_handler via the
existing myreq_handler_for_test seam: not-primary rejection (proven via
a poison-pointer sched.ep that would crash if forwarding were ever
attempted), not-connected rejection, and a genuine two-thread round-trip
test exercising the real signal/demux path.
Controller suite 3/3 (1090/1090 assertions), scheduler suite 26/26
unaffected, clang-format clean.
Build the live 3-node chaotic-multi-kill config-upgrade soak scenario
(run-ha-soak.sh --config-upgrade) required by Requirements 4.1, 4.3, 6.1,
6.4. This needed genuinely new client-side infrastructure: a hand-rolled
capnp send path for OP_upgradeConfig (the auto-generated encoder is a
no-op for its Data fields, worked around the same way
xce_replconfigpush_req_raw already does), plus cmd_hostctrl/cmd_upgradeconfig
CLI commands.
Building and running this live surfaced two real production bugs, both
fixed here:
- bug-1468: an unrelated prior commit inverted OPCode_upgradeConfig's
primary/standby dispatch guard in cluster.c, making config upgrades
unreachable from any network client.
- bug-1469: apply_committed_entry's WAL_CONFIG_MARKER case never called
config_activate_committed(), so the node that originates a config
upgrade never refreshed its own in-memory state as part of committing
it. Killing that node before some later, unrelated event happened to
refresh it left its checkpoint permanently stale -- the first live,
empirical reproduction of bug-857's originally hypothesized failure
mode. Fixed by activating before the marker-triggered checkpoint write,
since checkpoint content is built from live runtime tables that only
reflect the reload after activation runs.
A separate, narrower checkpoint divergence (bug-1471, not config-related)
remains open and still blocks a clean soak PASS -- task 8.6 stays
in-progress until it's resolved.
Send the HA heartbeat as OP_nop + the existing HeartbeatMsg body carrying
{term, committed_lsn, committed_config_tag} (senderPriority unused). Generic
xce_nop_req untouched; C-side body support pre-existed (no typemap edit). Req 1.1
producer half. Round-trip test proves the 3 fields survive the wire; lib suite
0 failures under ASAN.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xce_replconfig_reply_raw() - primary sends reply with real bytes via endpoint
xce_replconfig_req_ssl() - standby sends request over raw SSL socket
xce_replconfig_reply_ssl_decode() - standby decodes reply from raw SSL
xce_upgrade_config_req_extract() - documents architectural limitation; returns -ENOTSUP
xce_upgrade_config_req_extract_raw() - extract bytes from raw packet for OP_upgradeConfig
Same bypass pattern as existing xce_replconfigpush_req_raw(): build the
capnp message directly via capn_new_list8 + capn_setv8, sidestepping the
encode_Data_ptr path that routes raw bytes through xce_data_t (a
discriminated cluster-data union that cannot carry raw byte arrays).
Unblocks handle_config_request() and do_one_config_fetch() wiring.
Task A research finding: xce_upgrade_config_req_extract(xce_request_t*)
is unimplementable because xce_decode() calls capn_free() before any
handler runs, and decode_Data_ptr converts the wire-format List(UInt8)
into a zero-filled xce_data_t. Task D must capture the raw io_packet_t
bytes before xce_decode tears them down, then call the _raw variant.
Add the full call chain for host control operations from the Go API
server through the C library to the controller:
- schema/controller.capnp: HostControl struct + hostctrlmsg @21
in Request.body union
- include/typemap.h: xce_hostctrl_t* hostctrlmsg in body union
(placed after @20 clrpreasonmsg, matching capnp ordinal)
- include/client.h, xceapi.h, request.h: declarations for
xcecli_hostctrl, xce_hostctrl, xce_hostctrl_req
- lib/client.c: xcecli_hostctrl (validate + delegate to xce_hostctrl)
- lib/xceapi.c: xce_hostctrl (validate ctx state + send + wait reply)
- lib/request.c: xce_hostctrl_req (validate hostname/empty/nil,
accept only HOST_OPEN/HOST_CLOSE, reject HOST_TAG/UNTAG)
- lib/helper.c: Request_body_hostctrlmsg cases in
xce_request_body() and xce_request_msg_name()
- controller/cluster.c: OPCode_hostcontrol dispatch
- controller/include/cluster.h: do_hostcontrol_op declaration
- controller/worker.c: do_hostcontrol_op + fix pre-existing
goto-clean bug in worker_ctrl that leaked write lock and
reference on HOST_OPEN/HOST_CLOSE no-op paths
TDD: 8 xce_hostctrl_req validation tests (null ep, null ctrl,
null/empty hostname, HOST_DO_NONE no-op, HOST_TAG/UNTAG reject,
negative action reject, HOST_OPEN/CLOSE accept via xce_request).
116/116 xcelib tests pass. CGO_ENABLED=0 Go backend tests pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
