lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: correct controller restart recovery — jobs, workers, and resource ordering

Browse source 
Four bugs fixed:

1. persist.c: array job WAL loop aborted after index [0] when replication
   returned -ENODEV (no standbys). Loop guard now uses the return value of
   persist_wal_sync_append() which is 0 on local-disk success regardless of
   replication status.

2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
   return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
   up the existing slot by name, updates its endpoint, and clears UNAVAIL so
   the normal hook/reply/on_worker_join flow proceeds.

3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
   connection dropped because no error handler was registered. Added
   handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
   cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
   checking_connection() to detect the failure and rejoin.

4. cluster.c: persist_recover() was called before cluster_build_resources()
   populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
   fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
   so resmgr is fully populated before WAL replay.
This commit is contained in:
Rongsong Shen 2026-05-09 03:02:39 +00:00
parent b830cecf97
commit 6429ef0bff
4 changed files with 59 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

24
src/lib/xcectx.c
View file

@ -133,6 +133,29 @@ static int handle_msg(endpoint_t* ep)
(void*)ep);
}
/* Called by netio when the TCP connection drops (EPOLLHUP / EPOLLERR / read EOF).
* Unblocks any thread stuck in cbuf_get (e.g. xce_hello waiting for a reply)
* so that checking_connection can detect the failure and attempt to rejoin. */
static int handle_disconnect(endpoint_t* ep)
{
if (ep == NULL)
return 0;
xce_ctx_t* ctx = (xce_ctx_t*)ep->owner;
if (ctx == NULL)
return 0;
ctx->state = XCE_CTX_ERR;
/* Inject an error into wbuf so any blocked cbuf_get returns immediately.
* cbuf_tryput returns -EAGAIN if the buffer already has an item — that is
* fine: the existing item will wake the waiter. */
int err = -ECONNRESET;
cbuf_tryput(&ctx->wbuf, &err);
return 0;
}
int xce_ctx_init(xce_ctx_t* ctx, char* name, char* key)
{
int rc;
@ -237,6 +260,7 @@ int xce_ctx_start(xce_ctx_t* ctx)
endpoint_set_owner(ctx->ep, ctx);
endpoint_set_msg_handler(ctx->ep, (msg_handler_t)handle_msg);
endpoint_set_err_handler(ctx->ep, (err_handler_t)handle_disconnect);
rc = iomgr_monitor(ctx->ep);
if (rc < 0) {
ctx->state = XCE_CTX_ERR;