lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix: correct controller restart recovery — jobs, workers, and resource ordering

Browse source 
Four bugs fixed:

1. persist.c: array job WAL loop aborted after index [0] when replication
   returned -ENODEV (no standbys). Loop guard now uses the return value of
   persist_wal_sync_append() which is 0 on local-disk success regardless of
   replication status.

2. join.c: WAL-recovered worker slot (status=UNAVAIL) caused add_worker() to
   return -EEXIST on reconnect. New EEXIST path in handle_worker_join() looks
   up the existing slot by name, updates its endpoint, and clears UNAVAIL so
   the normal hook/reply/on_worker_join flow proceeds.

3. xcectx.c: cbuf_get() in xce_hello() blocked indefinitely when the TCP
   connection dropped because no error handler was registered. Added
   handle_disconnect() which sets ctx->state=XCE_CTX_ERR and calls
   cbuf_tryput(-ECONNRESET) to unblock the waiter, allowing
   checking_connection() to detect the failure and rejoin.

4. cluster.c: persist_recover() was called before cluster_build_resources()
   populated resmgr, causing restore_worker() → add_hostattrs_resvals() to
   fail with EEXIST noise. Moved persist_recover() to after HOOK_AFTER_EVENT
   so resmgr is fully populated before WAL replay.
This commit is contained in:
Rongsong Shen 2026-05-09 03:02:39 +00:00
parent b830cecf97
commit 6429ef0bff
4 changed files with 59 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

8
src/controller/persist.c
View file

@ -1016,10 +1016,14 @@ int persist_job_hook_fun(int type, int situation, int detail,
rc = -ENOMEM;
break;
}
/* persist_wal_sync_append returns 0 on successful local WAL write
* even when replication is degraded (no standbys → -ENODEV, timeout
* → -ETIMEDOUT). Only abort the loop on a real disk-write failure
* (non-zero return). Replication status is informational here. */
int sync_rc = 0;
persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz, &sync_rc);
rc = persist_wal_sync_append(p, WAL_JOB_SUBMIT, buf, (uint32_t)sz,
&sync_rc);
free(buf);
if (sync_rc != 0) rc = sync_rc;
}
return rc;
}