feat: acknowledged job start/finish/fail reports (bug-1741)

A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.

- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
  answered. A separate opcode because replies are matched through the
  client's single wbuf, not by sequence number: answering plain notifies
  would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
  do_notify, then waits on the commit barrier and replies 0 only once it
  is majority-committed. A barrier failure (the fact is still appended and
  retained) replies -E_XCE_NOT_READY, so the worker retries in place
  instead of rotating away from a healthy primary (cf. bug-1850). Admitted
  by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
  status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
  xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
  (a controller predating the opcode) they fall back to the plain
  fire-and-forget notify, so workers and controllers can be upgraded in
  either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
  yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
  retry budget (the rejoin resync remains the backstop). These calls run
  on the worker daemon's IPC callback, so while the controller lacks
  quorum other jobs' IPC messages wait (buffered, not lost) up to the
  retry budget.

Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-27 06:47:47 +00:00
parent cb1415d7c1
commit 0cad68507f
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
11 changed files with 236 additions and 6 deletions

View file

@ -24,6 +24,8 @@ extern int xce_update_rows(xce_ctx_t*, int, int, xlist_t*);
extern int xce_resval(xce_ctx_t*, char*, char*, xce_resval_t***);
extern int xce_notify(xce_ctx_t*, xce_notify_data_t*);
/* bug-1741: send a notify and wait for the controller's commit ACK. */
extern int xce_notify_acked(xce_ctx_t*, xce_notify_data_t*);
extern int xce_update_resvals(xce_ctx_t*, int, xce_resval_t**);
extern int xce_jobctrl(xce_ctx_t*, xce_jobctrl_t*);
extern int xce_queuectrl(xce_ctx_t*, xce_queuectrl_t*);