lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(scheduler): record config pushes, free hlimits lists, quiet benign picker WARN (bug-1847/1848/1849)

Browse source 
Follow-ups from the bug-1846 investigation.

bug-1847: config pushes never reached a .replay recording -- handler.c
applied them synchronously, bypassing the pending-update queue the
recorder writes from. With bug-1846's deferral they're now applied at a
round boundary on the driver thread, so sched_apply_pending_config()
writes a CONFIG_PUSH event via the existing sched_record_write_events()
(no-op when not recording), ahead of the round's other recorded events.
Test: test_config_push_deferred.c's test_applied_config_push_is_recorded
(RED: 0 events; GREEN: 1). Fidelity limit, pre-existing: replay applies
configPush through core_events.c's CONFIG_PUSH path, which doesn't re-run
groupmgr_load_configs()/plugin_config_update(), so a limits/groups change
still won't take effect in sim.

bug-1848: xce_sched_obj_t.hlimits was created with a NULL zarray killer
at all three constructors, so reset_sched_obj() and schedobj teardown
leaked the per-worker xlist_t match_job_host_limit_() callocs. New
kill_hlimits_list() (schedobj.c) frees the list, never its vres entries
(owned by the limit plugin); core_events.c and snapshot_load.c use
identical static local copies, since both avoid sharing helpers with
schedobj.c and several test binaries link them without it. Test:
test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED:
killer NULL; GREEN).

bug-1849: every --sim run warned "failed to resolve job picker plugin ...
picker rebuild will be skipped". Benign: the snapshot is loaded before
the sim registers its picker, and sim_bootstrap_picker() replays the same
queue/job walk afterward. Downgraded to TRACE with an explanation.

Buglog also records bug-1850 (open): the ~100s reconnect churn behind
bug-1846 part 2 -- xcecli_retry_next() rotates/reconnects on every retry
(ETIMEDOUT after the 10s reply deadline, E_XCE_NOT_READY, redirects), so
a slow reply from a healthy primary forces a rejoin + full re-push.
Needs the live scheduler.log and a libxce spec.

Full scheduler suite 87/87 (asan). Replay of the captured file: same 184
mismatches, all attributable to the recording's pre-fix binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-09-24 07:00:07 +00:00
parent a9feded88e
commit ac033f4783
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
8 changed files with 228 additions and 8 deletions
Download patch file Download diff file Expand all files Collapse all files

59
.wolf/buglog.json
View file

@ -25111,6 +25111,65 @@
],
"occurrences": 1,
"last_seen": "2026-09-23T10:05:00Z"
},
{
"id": "bug-1847",
"timestamp": "2026-09-24T05:30:00Z",
"error_message": "bug-1846 follow-up: a scheduler config push (sent by the controller on every scheduler join) was never written to a .replay recording, so a replay spanning a rejoin or a genuine config change never saw it.",
"file": "src/scheduler/src/scheduler.c (sched_apply_pending_config)",
"root_cause": "handler.c applied config pushes synchronously via sched_init_config(), bypassing the pending-update queue that xce_sched_driver_translate_pending_events() records from. bug-1846's deferral moved application to a round boundary on the driver thread (main.c -> sched_apply_pending_config()), making it recordable at the right point.",
"fix": "FIXED (2026-09-24, TDD). sched_apply_pending_config() now writes a CONFIG_PUSH core event via the existing sched_record_write_events() after sched_init_config() succeeds (no-op when not recording), ahead of the round's other recorded events -- matching the order they take effect. Test: test_config_push_deferred.c's test_applied_config_push_is_recorded (RED: record_event_count stayed 0; GREEN: 1 event written, stashing alone writes nothing). Known fidelity limit: replay applies configPush through core_events.c's CONFIG_PUSH path (queues/resources/resvals/configver), which does not re-run groupmgr_load_configs()/plugin_config_update()/sched_init_sharedInst() like sched_init_config() does -- a replayed config CHANGE to limits/groups still won't take effect in sim (pre-existing driver-equivalence gap, not introduced here).",
"tags": [
"scheduler",
"recording",
"replay",
"config-push",
"driver-equivalence"
]
},
{
"id": "bug-1848",
"timestamp": "2026-09-24T05:30:00Z",
"error_message": "Leak: xce_sched_obj_t.hlimits was created with a NULL zarray item killer, so reset_sched_obj() (limit_job_election()'s config-version rebuild) and schedobj teardown leaked the per-worker xlist_t that match_job_host_limit_() callocs, one per worker a job had tried. Visible as part of the large LeakSanitizer report on every --sim replay run.",
"file": "src/scheduler/src/schedobj.c, core_events.c, snapshot_load.c (hlimits zarray_init)",
"root_cause": "zarray_fini() only frees items through the zarray's killer; all three schedobj constructors passed NULL.",
"fix": "FIXED (2026-09-24, TDD). New kill_hlimits_list() (schedobj.c, declared in sched/types.h) frees one per-worker list (xlist_fini + free; never its vres entries, owned by the limit plugin's instance table); installed at all three zarray_init(&(obj->hlimits), ...) sites -- schedobj.c uses kill_hlimits_list(); core_events.c and snapshot_load.c get identical static local copies (core_hlimits_list_killer / snapshot_hlimits_list_killer) because both deliberately don't share helpers with schedobj.c and several test binaries (sched_core_events_test, sched_core_full_cycle_test, sched_snapshot_save_load_test, sched_sim_startup_test) link them without schedobj.c -- a shared symbol broke those links. zarray_fini() preserves the killer, so reset_sched_obj() leaves the zarray reusable. Test: test_limit_host_dedup.c's test_hlimits_zarray_releases_its_lists (RED: killer NULL; GREEN). A direct __lsan_do_recoverable_leak_check() test was tried and rejected -- the fixture's own resreq parsing (parse_select strdup) leaks independently, so the baseline is never leak-free.",
"tags": [
"scheduler",
"limit",
"hlimits",
"memory-leak"
]
},
{
"id": "bug-1849",
"timestamp": "2026-09-24T05:30:00Z",
"error_message": "Every --sim run logged 'WARN xce_sched_core_load_snapshot failed to resolve job picker plugin (rc=-2); picker rebuild will be skipped for this load' -- misleading; flagged as a suspect during the bug-1846 investigation.",
"file": "src/scheduler/src/snapshot_load.c (xce_sched_core_load_snapshot)",
"root_cause": "Benign ordering, not a fault: the only caller (sim.c's sim_locate_and_load_snapshot()) loads the snapshot before any picker is registered; sim_bootstrap_picker() (sim_picker.c, bug-1567) then replays the full on_queue_update + on_job_push walk against the restored core (builtin_fcfs has no on_config, and on_job_push filters non-PEND jobs itself), so the picker ends up fully rebuilt.",
"fix": "FIXED (2026-09-24). Downgraded to TRACE with a comment explaining the deferral; header comment updated. No test (log-level-only change).",
"tags": [
"scheduler",
"sim",
"snapshot",
"logging"
]
},
{
"id": "bug-1850",
"timestamp": "2026-09-24T05:30:00Z",
"error_message": "OPEN, NOT FIXED -- root trigger of bug-1846 part 2's mid-round rejoins: the live scheduler reconnected/re-joined roughly every ~100s in scheduler-replay-20260923-085356.replay (each rejoin = full config/worker/job re-push from on_scheduler_join()).",
"file": "src/lib/client_failover.c (xcecli_retry_next / xce_failover_classify), src/lib/xceapi.c (10s reply deadline)",
"root_cause": "UNCONFIRMED (code reading only; the live scheduler.log for the capture window isn't available here). The controller never closes a scheduler connection proactively (only handle_disconnect() on a socket error sets XCE_CTX_ERR). But xcecli_retry_next() calls xce_failover_rotate() -> xce_ctx_reconnect() on EVERY retry, and xce_failover_classify() treats ETIMEDOUT (any request whose reply takes >10s, cbuf_deadline_10s in xceapi.c), E_XCE_NOT_READY and E_XCE_REDIRECT/STANDBY as retryable. So a slow or 'not ready' reply to a dispatch/pend-reason publish from a HEALTHY primary rotates the scheduler to the next master (typically a standby, which redirects back), and each reconnect fires the rejoin callback -> sched_check_pending_rejoin() -> xcecli_join() -> full re-push, which adds load and can self-reinforce. Also: a reply arriving after its 10s deadline lands in the capacity-1 wbuf and can be consumed by the NEXT request (xcecli_ctx_recover() drains only one stale entry).",
"fix": "NOT FIXED. Next step: grep the live scheduler.log for 'publish retry-budget exhausted', 'xcecli failover', 'pending rejoin succeeded' around each truncated round to identify the triggering rc. Any fix changes libxce HA client failover semantics (e.g. not rotating away from a primary on ETIMEDOUT/NOT_READY) -> spec workflow per CLAUDE.md, not a drive-by.",
"tags": [
"libxce",
"client",
"failover",
"reconnect",
"ha",
"scheduler"
]
}
]
}