lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/src/controller/meson.build
Rongsong Shen 3eba7c16bf
fix(controller): fail loud on scheduler/worker disconnect instead of relying on the reply-wait timeout (bug-1693) 
Violates this project's own recorded Decision Log entry: "No client
reply-wait timeout as a fix for a hung request -- fail loud at the real
transport/source instead of masking with a deadline backstop."

OPCode_schedAdmin (cluster.c) and on_fetchfile (job.c, the worker proxy
-- cluster.c's own comment says schedAdmin "mirrors on_fetchfile's proxy
pattern", confirming one shared design mistake, not two independent
bugs) both blocked on a multi-second pthread_cond_timedwait, replying
"timeout" on expiry. handle_scheduler_leave()/handle_worker_leave() (the
actual disconnect handlers) only nulled the endpoint pointer -- neither
ever signaled the pending waiter, so a mid-wait disconnect was invisible
until the deadline lapsed. The timeout was the ONLY failure-detection
mechanism, exactly what the decision forbids.

Both sites already had a working signal function on their success path
(schedadmin_signal(), fetchfile_signal()) -- just never wired to the
disconnect handlers. Fixed by calling them from
handle_scheduler_leave()/handle_worker_leave() with a distinguishable
-ECONNRESET outcome, so the waiter wakes immediately on disconnect
instead of only on the deadline. Timeouts kept as defensive backstops
for a genuinely wedged peer that never disconnects cleanly either --
this fix is about not relying SOLELY on the timeout, not removing it.

schedadmin_signal_disconnect() must be called BEFORE handle_scheduler_leave()
acquires c->sched.lock -- the dispatch wait holds that same lock for its
entire duration, so signaling after acquiring it would deadlock.

Found by an /xce-audit-invariants controller pass (Decision Log fidelity
dimension). New test_fetchfile_disconnect.c plus extensions to
test_sched_admin_dispatch.c/test_sched_worker_change.c; RED/GREEN
independently verified against real elapsed-time assertions (pre-fix:
5001ms full-timeout wait; post-fix: <1s immediate wake). Full controller
suite: 1181 tests, 2 failures -- both the pre-existing, unrelated
bug-1561 overlayfs sandbox flake, confirmed via a clean meson-driven run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg
2026-08-25 14:13:26 +00:00

354 lines
13 KiB
Meson
Raw Blame History

controller_src = [
'cluster.c',
'commit.c',
'config_repl.c',
'ctrlstatus.c',
'dataex.c',
'egress.c',
'ha.c',
'ha_convergence.c',
'job.c',
'join.c',
'main.c',
'misc.c',
'myapp.c',
'opget.c',
'opupd.c',
'queue.c',
'reply.c',
'resources.c',
'sched.c',
'startup_validation.c',
'task.c',
'worker.c',
'apiserver.c',
'myconf.c',
'persist.c',
'repl.c',
'ctrl_metrics.c',
'slowlane.c',
]
incdir = include_directories('.', '../include', '../lib',
'include')
# cc = meson.get_compiler('c')
# math_dep = cc.find_library('m', required : false)
zlib_dep = dependency('zlib')
controller_deps = [
libxce_dep,
librtti_dep,
c_capnproto_dep,
math_dep,
tinylog_dep,
thread_dep,
zlib_dep,
]
controller = executable('xce_controller',
controller_src,
include_directories: incdir,
dependencies: controller_deps,
link_args: wrap_link_args,
install: true)
xce_test_src = [ 'testsuite/test.c', 'testsuite/waldump.c', 'testsuite/walcompare.c', 'testsuite/snapverify.c', 'testsuite/snapdiff.c', 'testsuite/configcompare.c', 'testsuite/resval_typeresolve.c']
# BSD-licensed readline replacement (scheduler-core spec, task 12.1 review
# finding): GNU readline is GPL; xce_test's REPL only needs readline()/
# add_history(), both provided by libedit's editline/readline.h emulation
# header under a permissive license. Single dependency, no separate
# ncurses/history split (libedit's own .pc pulls ncurses in privately).
libedit_dep = dependency('libedit')
ssl_dep = dependency('openssl') # also needed by configcompare.c (SHA256());
# controller_unit_test_deps below re-assigns
# the same dependency() result, harmless
xce_test_deps = [
libxce_dep,
librtti_dep,
struct2json_dep,
libedit_dep,
tinylog_dep,
zlib_dep, # waldump: crc32 WAL validation
c_capnproto_dep, # waldump: decode JOB_SUBMIT jobid
ssl_dep, # configcompare: SHA256() active-config hashing
]
xce_test = executable('xce_test',
xce_test_src,
include_directories: [incdir],
dependencies: xce_test_deps,
install: true)
# Controller Unity test suite (persist types and future controller unit tests)
controller_unit_test_src = [
'testsuite/controller_test.c',
'testsuite/test_persist_types.c',
'testsuite/test_persist_wal.c',
'testsuite/test_persist_schema.c',
'testsuite/test_persist_checkpoint.c',
'testsuite/test_persist_recovery.c',
'testsuite/test_hooks.c',
'testsuite/test_persist_hooks.c',
'testsuite/test_gc_retention.c',
'testsuite/test_cluster_persist.c',
'testsuite/test_e2e_wal.c',
'testsuite/test_e2e_checkpoint.c',
'testsuite/test_e2e_hooks.c',
'testsuite/test_e2e_gc.c',
'testsuite/test_e2e_capnp.c',
'testsuite/test_ha_role.c',
'testsuite/test_ha_role_lifecycle.c',
'testsuite/test_ha_cluster.c',
'testsuite/test_ha_redirect.c',
'testsuite/test_join_redirect.c',
'testsuite/test_p1_integration.c',
'testsuite/test_repl_unit.c',
'testsuite/test_repl_apply.c',
'testsuite/test_persist_repl.c',
'testsuite/test_repl_catchup.c',
'testsuite/test_p2_integration.c',
'testsuite/test_p3_hardening.c',
'testsuite/test_p3_manual_failover.c',
'testsuite/test_ctrl_metrics.c',
'testsuite/test_config_repl.c',
'testsuite/test_worker_join.c',
'testsuite/test_ha_startup.c',
'testsuite/test_ha_election.c',
'testsuite/test_ha_elect_ctx_lifecycle.c',
'testsuite/test_ha_bootstrap.c',
'testsuite/test_ha_bootstrap_support.c',
'testsuite/test_ha_durability.c',
'testsuite/test_repl_majority.c',
'testsuite/test_persist_truncate.c',
'testsuite/test_ha_ctrlhello.c',
'testsuite/test_ha_failover.c',
'testsuite/test_ha_vote_send.c',
'testsuite/test_ha_start_election_async.c',
'testsuite/test_ha_vote_gate.c',
'testsuite/test_ha_repl_push.c',
'testsuite/test_ha_connection_reuse.c',
'testsuite/test_ha_rdhup_integration.c',
'testsuite/test_ha_liveness.c',
'testsuite/test_ha_prevote.c',
'testsuite/test_ha_failover_prevote.c',
'testsuite/test_commit_engine.c',
'testsuite/test_commit_barrier_rollout.c',
'testsuite/test_commit_barrier_reply.c',
'testsuite/test_egress_filter.c',
'testsuite/test_demotion_convergence.c',
'testsuite/test_convergence_verification.c',
'testsuite/test_coldstart_lsn0.c',
'testsuite/test_commit_durability.c',
'testsuite/test_ctrlstatus.c',
'testsuite/test_ha_heartbeat_committed.c',
'testsuite/test_ha_vote_floor_heartbeat.c',
'testsuite/test_term_safety_adapter.c',
'testsuite/test_drive_async_config.c',
'testsuite/test_slowlane.c',
'testsuite/test_commit_admission_pool.c',
'testsuite/test_commit_resolve.c',
'testsuite/test_ha_vep_teardown.c',
'testsuite/test_ha_dispatch_integration.c',
'testsuite/test_term_adoption_vectors.c',
'testsuite/test_ha_core_safety.c',
'testsuite/test_ha_partition.c',
'testsuite/test_waldump.c',
'testsuite/waldump.c',
'testsuite/test_walcompare.c',
'testsuite/walcompare.c',
'testsuite/test_snapverify.c',
'testsuite/snapverify.c',
'testsuite/test_snapdiff.c',
'testsuite/snapdiff.c',
'testsuite/test_configcompare.c',
'testsuite/configcompare.c',
'testsuite/test_persist_snapshot_entities.c',
'testsuite/test_apply_seam.c',
'testsuite/test_apply_committed_entry.c',
'testsuite/test_replay_apply_seam.c',
'testsuite/test_wal_job_dispatch_fact.c',
'testsuite/test_wal_job_start_fact.c',
'testsuite/test_wal_job_finish_fact.c',
'testsuite/test_wal_job_rusage_fact.c',
'testsuite/test_wal_job_ckpt_fact.c',
'testsuite/test_wal_job_preason_fact.c',
'testsuite/test_job_dispatch_fail_count.c',
'testsuite/test_apply_reapply_idempotency.c',
'testsuite/test_apply_correctness_verification.c',
'testsuite/test_apply_pump.c',
'testsuite/test_apply_pump_integration.c',
'testsuite/test_job_array_apply.c',
'testsuite/test_job_array_fence.c',
'testsuite/test_job_array_replay_fence.c',
'testsuite/test_job_dispatch.c',
'testsuite/test_dispatch_commit.c',
'testsuite/test_dispatch_fail_terminal.c',
'testsuite/test_dispatch_fail_retry.c',
'testsuite/test_jobdispatch_no_double_notify.c',
'testsuite/test_upgrade_snapshot_restore.c',
'testsuite/test_upgrade_snapshot_apply.c',
'testsuite/test_bug857_regression.c',
'testsuite/test_sched_admin_dispatch.c',
'testsuite/test_fetchfile_disconnect.c',
'ctrlstatus.c',
'commit.c',
'job.c',
'egress.c',
'ha_convergence.c',
'queue.c',
'join.c',
'opupd.c',
'startup_validation.c',
'worker.c',
'persist.c',
'ha.c',
'repl.c',
'ctrl_metrics.c',
'config_repl.c',
'slowlane.c',
'cluster.c',
]
ssl_dep = dependency('openssl')
controller_unit_test_deps = [
libxce_dep,
librtti_dep,
unity_dep,
tinylog_dep,
thread_dep,
zlib_dep,
c_capnproto_dep,
ssl_dep,
]
controller_unit_test = executable('controller_unit_test',
controller_unit_test_src,
include_directories: [incdir],
dependencies: controller_unit_test_deps,
c_args: ['-DXCE_TESTING'],
install: false)
test('controller test', controller_unit_test,
suite: 'controller',
timeout: 300,
env: ['ASAN_OPTIONS=detect_leaks=0'])
# job-array-single-wal-record task 5.1: sched.c's push logic (sched_job_new_fun /
# xcesched_push_jobarray) is exercised by a small, STANDALONE test binary rather
# than being folded into controller_unit_test_src above. sched.c/dataex.c are not
# part of that binary today -- several existing files there (test_commit_barrier_
# reply.c, test_worker_join.c) already provide their OWN local fake definitions
# of on_scheduler_join()/notify()/collect_confvars(), precisely because the real
# sched.c/dataex.c were never linked in. Linking the real ones alongside those
# fakes collides at link time (multiple definition). This binary links the real
# sched.c + its real dataex.c/egress.c dependencies, isolated from that
# collision; only opget.c's one-line check_worker_() is stubbed locally in the
# test file (linking all of opget.c would pull in its own further dependencies
# this test does not need).
sched_push_unit_test_src = [
'testsuite/test_job_array_sched_push.c',
'sched.c',
'dataex.c',
'egress.c',
]
sched_push_unit_test = executable('sched_push_unit_test',
sched_push_unit_test_src,
include_directories: [incdir],
dependencies: controller_unit_test_deps,
c_args: ['-DXCE_TESTING'],
install: false)
test('sched push test', sched_push_unit_test,
suite: 'controller',
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])
# bug-1578 regression test: sched_worker_change()'s dangling-pointer fix
# (w->host.resvals_to_sched/nresvals_to_sched left aliased onto a freed
# xlist_t backing array -- see sched.c's sched_worker_change() for the full
# history). Same real-vs-fake link-collision reason as sched_push_unit_test
# just above requires its own standalone binary rather than folding into
# controller_unit_test_src: sched_worker_change() is only reachable by
# linking the REAL sched.c/dataex.c/egress.c, which collides with other
# controller_unit_test_src files' own fake on_scheduler_join()/notify()
# definitions. Mirrors sched_push_unit_test_src's exact source-set strategy
# (including opget.c's check_worker_() being stubbed locally rather than
# linking all of opget.c).
sched_worker_change_unit_test_src = [
'testsuite/test_sched_worker_change.c',
'sched.c',
'dataex.c',
'egress.c',
]
sched_worker_change_unit_test = executable('sched_worker_change_unit_test',
sched_worker_change_unit_test_src,
include_directories: [incdir],
dependencies: controller_unit_test_deps,
c_args: ['-DXCE_TESTING'],
install: false)
test('sched worker change test', sched_worker_change_unit_test,
suite: 'controller',
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])
# job-array-single-wal-record task 5.2: apiserver.c's push logic
# (apiserver_job_new_fun / apiserver_push_jobarray) faces the identical
# real-vs-fake link collision task 5.1 hit for sched.c (test_commit_barrier_
# reply.c / test_worker_join.c already define their OWN fake notify()/
# on_scheduler_join(), so linking the real dataex.c into controller_unit_test
# collides at link time) -- so this is its own standalone binary too,
# mirroring sched_push_unit_test's strategy. UNLIKE sched.c, apiserver.c
# never calls job_submit_committed()/collect_jobs_committed() (egress.c), so
# this binary does NOT need egress.c -- verified by grep, not assumed
# identical to sched.c's dependency set.
apiserver_push_unit_test_src = [
'testsuite/test_job_array_apiserver_push.c',
'apiserver.c',
'dataex.c',
]
apiserver_push_unit_test = executable('apiserver_push_unit_test',
apiserver_push_unit_test_src,
include_directories: [incdir],
dependencies: controller_unit_test_deps,
c_args: ['-DXCE_TESTING'],
install: false)
test('apiserver push test', apiserver_push_unit_test,
suite: 'controller',
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])
# resource-value-verification task 1.1: resval_typeresolve.c is a small,
# dependency-light module (no networking/REPL/global state) extracted from
# test.c's set_resval() specifically so its name -> Value_u_which resolution
# logic can be unit-tested in isolation -- test.c itself (4298 lines, its own
# main(), heavy libedit/network dependencies) cannot be white-box included
# into a Unity test. Standalone binary (not folded into controller_unit_test_
# src) since it has zero dependency on controller.c/cluster.c internals,
# mirroring sched_push_unit_test's/apiserver_push_unit_test's own standalone-
# binary rationale above.
resval_typeresolve_unit_test_src = [
'testsuite/test_resval_typeresolve.c',
'testsuite/resval_typeresolve.c',
]
resval_typeresolve_unit_test = executable('resval_typeresolve_unit_test',
resval_typeresolve_unit_test_src,
include_directories: [incdir],
dependencies: controller_unit_test_deps,
c_args: ['-DXCE_TESTING'],
install: false)
test('resval typeresolve test', resval_typeresolve_unit_test,
suite: 'controller',
timeout: 60,
env: ['ASAN_OPTIONS=detect_leaks=0'])
Reference in a new issue View git blame Copy permalink