feat: acknowledged job start/finish/fail reports (bug-1741)

A worker reported job start, finish and failure with a fire-and-forget
OPCode_notify the controller never answers, so a report lost while the
worker stayed connected (the live incident: 6 of 7 finish reports in a
burst) left the job RUN forever; the rejoin resync (165fc158) only
recovers reports lost around a reconnect.

- controller.capnp: OPCode_notifyAcked -- the same NotifyData body,
  answered. A separate opcode because replies are matched through the
  client's single wbuf, not by sequence number: answering plain notifies
  would feed stale statuses to the next synchronous request.
- controller (cluster.c): do_notify_acked() applies the fact like
  do_notify, then waits on the commit barrier and replies 0 only once it
  is majority-committed. A barrier failure (the fact is still appended and
  retained) replies -E_XCE_NOT_READY, so the worker retries in place
  instead of rotating away from a healthy primary (cf. bug-1850). Admitted
  by the no-quorum fact-bearing gate like OPCode_notify.
- libxce: xce_notify_acked_req/xce_notify_acked (send, wait for the
  status) and xcecli_notify_acked (XCECLI_FAILOVER_CALL retry/failover).
  xcecli_notify_jobstart/jobfinish/jobfail use it; on -E_XCE_BAD_OPCODE
  (a controller predating the opcode) they fall back to the plain
  fire-and-forget notify, so workers and controllers can be upgraded in
  either order.
- Resends are safe: START/FINISH are duplicate-guarded and a repeated FAIL
  yields the same state.
- worker: logs a WARN when a report is still unacknowledged after the
  retry budget (the rejoin resync remains the backstop). These calls run
  on the worker daemon's IPC callback, so while the controller lacks
  quorum other jobs' IPC messages wait (buffered, not lost) up to the
  retry budget.

Tests: test_notify_acked_reply_only_after_commit and
test_notify_acked_withheld_replies_not_ready (no reply before the covering
ACK; -E_XCE_NOT_READY with the fact retained when ACKs are withheld).
Written with the opcode, so no separate RED run. Full /xce-build all:
111 suites pass; controller suite 1245/0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

src/lib/client.c

@@ -562,7 +562,7 @@ int xcecli_notify_jobfinish(xcecli_ctx_t c, xce_jobfinish_t* jfinish)
         return -EINVAL;
     }
 
-    return xcecli_notify(ctx, &data);
+    return xcecli_notify_acked(ctx, &data); /* bug-1741 */
 }
 
 int xcecli_update_resvals(xcecli_ctx_t c, int nres, xce_resval_t** vals)
@@ -636,6 +636,30 @@ int xcecli_notify(xcecli_ctx_t c, xce_notify_data_t* data)
     return XCECLI_FF_CALL(ctx, xce_notify(&(ctx->ctx), data));
 }
 
+/* bug-1741: see client.h. A lost job start/finish/fail report used to be
+ * gone for good (fire-and-forget, no reply), leaving the job RUN forever
+ * unless the worker happened to rejoin. This resends until the controller
+ * confirms the fact committed (duplicates are harmless controller-side). An
+ * old controller answers the unknown opcode with -E_XCE_BAD_OPCODE: send
+ * the plain fire-and-forget notify instead, so worker and controller can be
+ * upgraded in either order. */
+int xcecli_notify_acked(xcecli_ctx_t c, xce_notify_data_t* data)
+{
+    api_ctx_t* ctx = (api_ctx_t*)c;
+    int        rc;
+
+    if ((c == NULL) || (data == NULL)) {
+        return -EINVAL;
+    }
+
+    rc = XCECLI_FAILOVER_CALL(ctx, /*is_write=*/true,
+                              xce_notify_acked(&(ctx->ctx), data));
+    if (rc == -E_XCE_BAD_OPCODE) {
+        rc = xcecli_notify(ctx, data);
+    }
+    return rc;
+}
+
 int xcecli_notify_jobstart(xcecli_ctx_t c, job_pidinfo_t* pidinfo)
 {
     api_ctx_t* ctx = (api_ctx_t*)c;
@@ -652,7 +676,7 @@ int xcecli_notify_jobstart(xcecli_ctx_t c, job_pidinfo_t* pidinfo)
         return -EINVAL;
     }
 
-    return xcecli_notify(ctx, &data);
+    return xcecli_notify_acked(ctx, &data); /* bug-1741 */
 }
 
 int xcecli_notify_jobfail(xcecli_ctx_t c, job_failinfo_t* failinfo)
@@ -671,7 +695,7 @@ int xcecli_notify_jobfail(xcecli_ctx_t c, job_failinfo_t* failinfo)
         return -EINVAL;
     }
 
-    return xcecli_notify(ctx, &data);
+    return xcecli_notify_acked(ctx, &data); /* bug-1741 */
 }
 
 int xcecli_jobctrl(xcecli_ctx_t c, xce_jobctrl_t* ctrlinfo)