fix(controller): close 6 data-safety races + apply-pump O(N^2) regression + boot TOCTOU

Bundles the fixes validated this session across ha.c/persist.c/worker.c:

- raft-apply-decoupling Task 7.4: apply-pump O(N^2) throughput regression
  fixed via a segment-read cache.
- bug-1416: boot-time TOCTOU race in persist_verify_apply_correctness()
  closed with a recovery-complete gate that closes at process-lifecycle
  start, not function entry.
- bug-451: HOOK_DETAIL_JOB_RUSAGE_UPDATE now has its own persist hook arm
  instead of silently falling through the default job-state case.
- bug-1426: trim_wal_segments() also caps its trim floor at the local
  apply pump's own progress, not just remote standby-acked positions.
- bug-1427: persist_verify_apply_correctness() extended with a worker-count
  check mirroring the existing job-count check (count_jobmgr_entries and
  count_workermgr_entries now share one count_manager_entries(var) helper).
- bug-1428: persist_register_pending_apply's critical section widened to
  close the actual race instead of patching a symptom.
- bug-1472 (remainder): WORKER_DEAD replay in worker_from_capnp() now
  applies mark_worker_unavail()'s mutations via a shared
  apply_worker_dead_mutation() helper instead of duplicating them.
- bug-1474 (angle-a): on_standby_connected's stale-duplicate detection
  fixed for promoted vote endpoints via a shared votes_idx_for_peer()
  index-scale converter.

Each fix independently verified via TDD (RED before fix, GREEN after).
Full ASAN rebuild + controller_unit_test: 1101 tests, 0 failures. Live
single-node HA deployment validated Task 7.4 and bug-1416 across 5
restart/recovery cycles (45/45 job submissions, zero APPLY_DIVERGENT
fences).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-08-16 10:20:02 +00:00
parent 3afe3d5549
commit 45400c3a28
Signed by: rshen
SSH key fingerprint: SHA256:8Vf2pZ3h6rJa7EhGK+WaAQ9Mta/apZvteW6g9U/WKKk
18 changed files with 1499 additions and 195 deletions

View file

@ -799,8 +799,12 @@ static int read_memory_info(double* mem_total, double* mem_free,
}
/**
* @brief The mem index gives the currently available physical memory in MB.
* @return A string representing the available physical memory in megabytes.
* @brief The mem index gives the currently available physical memory in
* bytes (canonical internal/wire unit -- same convention as the
* maxmem host attribute; CLI display converts to a human-readable
* unit, config-time thresholds may be written with a unit suffix
* like "500MB" and are converted to bytes when parsed).
* @return A string representing the available physical memory in bytes.
*/
char* node_info_get_mem(void)
{
@ -813,14 +817,14 @@ char* node_info_get_mem(void)
return NULL;
}
// Read memory information
// Read memory information (read_memory_info reports MB)
if (read_memory_info(&mem_total, &mem_free, &mem_used) != 0) {
free(result);
return NULL;
}
// Format the available memory as a string (in MB)
int ret = snprintf(result, 64, "%.2f", mem_free);
// Format the available memory as a string (in bytes)
int ret = snprintf(result, 64, "%.2f", mem_free * 1024.0 * 1024.0);
if (ret < 0 || ret >= 64) {
XCECD_LOG(XCE_LOG_ERROR, "Failed to format memory information string");
free(result);
@ -831,9 +835,11 @@ char* node_info_get_mem(void)
}
/**
* @brief The tmp index is the space available in MB or in units set
* on the file system that contains the temporary directory.
* @return A string representing the available space in temporary directory.
* @brief The tmp index is the space available in bytes (canonical internal/
* wire unit -- see node_info_get_mem) on the file system that
* contains the temporary directory.
* @return A string representing the available space in the temporary
* directory, in bytes.
*/
char* node_info_get_tmp(void)
{
@ -859,11 +865,8 @@ char* node_info_get_tmp(void)
unsigned long long free_bytes
= (unsigned long long)fs_stat.f_bavail * fs_stat.f_frsize;
// Convert to MB
double tmp_free_mb = (double)free_bytes / (1024.0 * 1024.0);
// Format the available tmp space as a string (in MB)
int ret = snprintf(result, 64, "%.2f", tmp_free_mb);
// Format the available tmp space as a string (in bytes)
int ret = snprintf(result, 64, "%llu", free_bytes);
if (ret < 0 || ret >= 64) {
XCECD_LOG(XCE_LOG_ERROR, "Failed to format tmp space string");
free(result);

View file

@ -348,8 +348,12 @@ static void init_host_attrs(void)
= hardware_monitor_get_total_memory(xcecd_ctx.hardware_monitor);
}
host_attrs.maxswap = 1024;
host_attrs.maxtmp = 2048;
/* Placeholder stub values (bug-1554, real swap/tmp-disk detection not
* yet implemented) -- kept as bytes to match every other memory-scale
* host attribute's canonical unit (maxmem, mem, tmp). Magnitude chosen
* to match the pre-existing MB-scale placeholders (1024MB/2048MB). */
host_attrs.maxswap = 1024.0 * 1024.0 * 1024.0;
host_attrs.maxtmp = 2048.0 * 1024.0 * 1024.0;
host_attrs.ndisks = 1;
host_attrs.hostmodel = strdup("Intel64");