forked from lei.chen/myxce
Rongsong Shen
b9ad01878c
Root-caused via raft-ha-core-safety 5.10a investigation: schedule_checkpoint() (persist.c) fires correctly once wal_record_count crosses its threshold, but its delay_task()-scheduled checkpoint_task_fn never executed in a live multi-node deployment -- confirmed live via temporary diagnostic logging (wal_record_count=100 max=100 logged, but the task itself never ran). Root cause: delay_task_t carried TWO incomparable time representations -- `nextrun` (wall-clock time_t seconds, used by delay_task()) and `nextrun_ts` (CLOCK_MONOTONIC timespec, used by delay_task_ms()). task_cmp()'s mixed-type fallback compared them directly: an ms-task's nextrun is always 0, a seconds-task's nextrun is a real epoch value (e.g. 1785600123) -- so 0 always sorted "less than" any seconds-task, regardless of actual due time. ordq (src/lib/ordq.c) keeps its queue as an ascending sorted linked list by this comparator, and delay_worker() only ever inspects the list HEAD (ordq_peek with no matchfn returns q->head immediately, no scanning). Since cluster_t.delaymgr is shared with ha.c's ~10 recurring delay_task_ms() call sites (heartbeats, election timers, RDHUP polling), the head was structurally almost-always occupied by an ms-task -- any delay_task() (seconds) caller sharing that delaymgr, including schedule_checkpoint's checkpoint_task_fn, could be starved indefinitely. Fix: eliminate the dual representation entirely. Every task -- whether scheduled via delay_task() (seconds) or delay_task_ms() (milliseconds) -- now stores one flat uint64_t CLOCK_MONOTONIC millisecond deadline (nextrun_ms). delay_task() becomes a thin seconds->ms wrapper around delay_task_ms(). task_cmp() and task_due() collapse to a single integer comparison; delay_worker()'s reschedule is a flat integer add (no more tv_sec/tv_nsec carry arithmetic, no more is_ms_task branching). Net -42 lines. Live-verified: after this fix, a forced checkpoint round produces a real installed snapshot file (previously: schedule_checkpoint fired but the task silently never ran). controller_unit_test: full 58-group sweep, 0 Failures, 0 ASAN reports (before and after this change) -- no regression, including every delaymgr-touching group (test_demotion_convergence, test_ha_start_election_async, test_ha_core_safety, test_ha_startup, etc). |
||
|---|---|---|
| .. | ||
| autoscaler | ||
| controller | ||
| docs | ||
| frontend | ||
| include | ||
| lib | ||
| nixSupport | ||
| scheduler | ||
| schema | ||
| tools | ||
| worker | ||
| flake.lock | ||
| flake.nix | ||
| meson.build | ||
| meson.options | ||