feat: acknowledged job start/finish/fail reports (bug-1741)

A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.

- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
  answered. A separate opcode because replies are matched through the
  client's single wbuf, not by sequence number: answering plain notifies
  would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
  do_notify, then waits on the commit barrier and replies 0 only once it
  is majority-committed. A barrier failure (the fact is still appended and
  retained) replies -E_XCE_NOT_READY, so the worker retries in place
  instead of rotating away from a healthy primary (cf. bug-1850). Admitted
  by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
  status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
  xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
  (a controller predating the opcode) they fall back to the plain
  fire-and-forget notify, so workers and controllers can be upgraded in
  either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
  yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
  retry budget (the rejoin resync remains the backstop). These calls run
  on the worker daemon's IPC callback, so while the controller lacks
  quorum other jobs' IPC messages wait (buffered, not lost) up to the
  retry budget.

Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

src/controller/cluster.c

@@ -124,6 +124,46 @@ static void handle_endpoint_error(endpoint_t* ep)
     return;
 }
 
+static int do_notify(cluster_t* c, endpoint_t* ep, xce_request_t* req);
+
+/* bug-1741: OPCode_notifyAcked -- the same job-lifecycle facts as
+ * OPCode_notify, but answered: the reply is sent only after the fact's WAL
+ * entry is majority-committed, so a worker that got status 0 knows the fact
+ * survives a failover, and a worker that got an error (or no reply) resends
+ * it. A resend is harmless: START/FINISH are guarded against duplicates and a
+ * repeated FAIL yields the same state. A barrier failure (e.g. transient
+ * quorum loss -- the fact itself is retained) is answered -E_XCE_NOT_READY
+ * so the worker retries in place rather than rotating away from a healthy
+ * primary (cf. bug-1850). */
+static int do_notify_acked(cluster_t* c, endpoint_t* ep, xce_request_t* req)
+{
+    xce_reply_t reply;
+    int         rc;
+
+    init_reply_by_req(&reply, req);
+
+    do_notify(c, ep, req);
+    rc = persist_commit_barrier(&c->persist, 0);
+    if (rc != 0) {
+        WARN("%s: commit barrier failed (rc %d) -- worker will resend",
+             __func__, rc);
+        rc = -E_XCE_NOT_READY;
+    }
+
+    reply.status     = rc;
+    reply.data_which = RequestReply_data_nodata;
+    xce_reply(ep, c->myhost, &reply);
+
+    return 0;
+}
+
+#ifdef XCE_TESTING
+int do_notify_acked_for_test(cluster_t* c, endpoint_t* ep, xce_request_t* req)
+{
+    return do_notify_acked(c, ep, req);
+}
+#endif
+
 static int do_notify(cluster_t* c, endpoint_t* ep, xce_request_t* req)
 {
     xce_notify_data_t* data = req->body.notifymsg;
@@ -612,6 +652,9 @@ static int myreq_handler(vtask_t* task)
     case OPCode_notify:
         rc = do_notify(c, ep, &request);
         break;
+    case OPCode_notifyAcked:
+        rc = do_notify_acked(c, ep, &request);
+        break;
     case OPCode_clusterconf:
         rc = handle_clusterconf(c, ep, &request);
         break;