lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: correct controller restart recovery — jobs, workers, and resource ordering

Browse source 
Four bugs fixed:

1. persist.c: array job WAL loop aborted after index [0] when replication
   returned -ENODEV (no standbys). Loop guard now uses the return value of
   persist_wal_sync_append() which is 0 on local-disk success regardless of
   replication status.

2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
   return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
   up the existing slot by name, updates its endpoint, and clears UNAVAIL so
   the normal hook/reply/on_worker_join flow proceeds.

3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
   connection dropped because no error handler was registered. Added
   handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
   cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
   checking_connection() to detect the failure and rejoin.

4. cluster.c: persist_recover() was called before cluster_build_resources()
   populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
   fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
   so resmgr is fully populated before WAL replay.
This commit is contained in:
Rongsong Shen 2026-05-09 03:02:39 +00:00
parent b830cecf97
commit 6429ef0bff
4 changed files with 59 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

17
src/controller/join.c
View file

@ -61,10 +61,25 @@ static int handle_worker_join(cluster_t* c, endpoint_t* ep,
HOOK_DETAIL_WORKER_JOIN, c, from->name, 0);
idx = add_worker(c, ep, from->name, joinmsg->data.attrs);
/* A worker recovered from WAL already occupies the slot (status=UNAVAIL).
* Reconnecting after a controller restart will fail add_worker with -EEXIST.
* Locate the existing slot, update its endpoint, and clear UNAVAIL. */
if (idx == -EEXIST) {
worker_t* existing = NULL;
if (key_find(c->workermgr, (void*)from->name, (void**)&existing) > 0 &&
existing != NULL) {
lock(existing, RTTI_LOCK_WRITE);
existing->ep = ep;
existing->host.status &= ~XCE_WORKER_UNAVAIL;
idx = existing->idx;
unlock(existing);
}
}
reply.status = idx;
reply.data_which = RequestReply_data_nodata;
/* Look up the newly added worker so the persist hook can serialize it */
/* Look up the worker so the persist hook can serialize it */
worker_t* w = NULL;
if (idx >= 0)
vget(c->workermgr, idx, (void**)&w);