No description
Find a file
Rongsong Shen a9feded88e
fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2)
sched_publish_retry_loop() calls sched_check_pending_rejoin() after every
successful publish -- inside the driver's decision-drain loop -- so after a
client-library reconnect the scheduler re-joins mid-round. The controller
answers every join (on_scheduler_join(), controller/sched.c) by re-pushing
config, workers and jobs. handler.c applied the config push synchronously
on the handler thread: sched_init_config() -> sched_init_queues() ->
builtin_fcfs on_queue_update(), which nulls the picker cursor when it
reinserts the queue the cursor rests on. next_job() then returned NULL and
the round ended with the rest of the queue never considered.

Evidence (scheduler-replay-20260923-085356.replay, temporary event-stream
diagnostics, reverted): 9 recorded rounds end after 11-12 decisions; every
one is followed by a workersPush + full 50-job jobsPush (the join's resync,
reporting just-dispatched jobs back as PEND), every full-length dispatching
round has none, and the config push never appears because it bypassed the
pending-update queue the recorder writes from -- which is also why replay
could not reproduce the truncation.

Fix (defer to the round edge, same direction as bug-1813): handler.c now
calls new sched_stash_config(), parking the Conf4Sched in new
xce_scheduler_t.pending_config under pending_config_lock (an older
unapplied push is freed). main.c's loop calls new
sched_apply_pending_config() before each round, running the unchanged
sched_init_config() strictly between sessions. sched_fini() frees a
never-applied push.

Tests:
- test_handlers.c: test_config_push_is_stashed_not_applied_on_handler_thread
  -- RED (sched_init_config called on the handler thread), GREEN after.
- new test_config_push_deferred.c (sched_config_push_deferred_test): real
  scheduler_req_handler() + real builtin_fcfs mid-session -- RED (next_job()
  returned NULL right after the push), GREEN (session walks the remaining
  jobs; configver unchanged until applied once between rounds).
- test_admin_request.c / test_job_array_expand.c / test_job_array_wire_e2e.c
  #include handler.c directly and get a link stub for sched_stash_config().
Full scheduler suite 87/87 (asan).

Not addressed: config pushes are still not written to a recording (a
replay spanning a config change won't see it); the ~100s reconnect churn
itself (see bug-1812/1813).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 06:30:34 +00:00
.claude test(xce-test-ha): add --worker-capacity-soak mode for bug-1841 regression 2026-09-22 13:14:44 +00:00
.kiro chore: persist OpenWolf state, controller-peers-env-fallback spec, and scheduler backlog notes 2026-09-23 03:19:52 +00:00
.wolf fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2) 2026-09-24 06:30:34 +00:00
docs/superpowers/plans update documents 2026-08-10 17:51:51 +08:00
issues chore: persist OpenWolf state, controller-peers-env-fallback spec, and scheduler backlog notes 2026-09-23 03:19:52 +00:00
raft chore: checkpoint all prior work for clean state 2026-06-04 13:04:16 +00:00
reports docs: add weekly report and change log for Sep 13-19, 2026 2026-09-20 13:18:15 +00:00
scripts fix(gitea-sync): unwrap buglog.json's {version,bugs} wrapper before iterating (bug-1677) 2026-08-24 14:23:31 +00:00
src fix(scheduler): mid-round rejoin config push truncated live rounds (bug-1846 part 2) 2026-09-24 06:30:34 +00:00
.clang-format reformat code and job array 2025-11-07 16:14:27 +08:00
.gitignore ignore daemon.log token-ledger 2026-08-28 14:33:03 +08:00
.mcp.json checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
.opencode.json checkpoint: bug-851 root cause + raft TLA+ conformance spec family 2026-06-21 14:02:15 +00:00
CHANGES.md docs: add weekly report and change log for Sep 13-19, 2026 2026-09-20 13:18:15 +00:00
CLAUDE.md docs(xce-harness-component): document harness in root CLAUDE.md and build-deployment.md 2026-09-14 07:45:26 +00:00
README.md chore(build): make xce-build.sh self-wrap into nix develop 2026-09-07 15:31:06 +00:00

1. commit 规则

  • 每个人 fork 自己的 branch, 从自己的 branch 做 PR,review 完再 commit

2. code format

2.1. C

推荐使用 clang-format,公共配置放在源码里面 xce/.clang-format

(理论上直接使用源码里面 xce/.clang-format 文件格式不会变)

2.1.1. vscode 可以在配置中加上如下内容

"C_Cpp.clang_format_style": "WebKit",
"C_Cpp.clang_format_fallbackStyle": "WebKit",

2.1.2. vim & emacs

clang-format -style="{BasedOnStyle: webkit}" -i XX.c

2.2. golang

使用默认 gofmt

3. build environment and build (example of ubuntu)

3.1 build environment

  • Install nix package management tool
sh <(curl -L https://nixos.org/nix/install) --daemon
  • Edit /etc/nix/nix.conf to make sure
experimental-features = nix-command flakes
sandbox = true
sandbox-fallback = false
substituters = https://mirrors.ustc.edu.cn/nix-channels/store https://cache.nixos.org https://cache.nixos.org/
trusted-users = root <your account>

make sure that you have restarted nix-daemon service after you have changed /etc/nix/nix.conf

  • Install socat and config ssh (for clone by ssh protocol)
nix profile add nixpkgs#socat

.ssh/config

Host gitea.shenrs.eu
	HostName gitea.shenrs.eu
	IdentityFile ~/.ssh/id_ed25519
	ProxyCommand socat - openssl-connect:gitea.shenrs.eu:2022,snihost=gitea.shenrs.eu,verify=0

3.2 checkout source code

git clone ssh://forgejo@gitea.shenrs.eu/rshen/xce.git
cd xce

(参考第 1 节,fork 自己的 branch 再提交 PR)

3.3 build (for package and release)

cd xce/src

# for build debug version
nix build .#xcecore_debug

# for normal build
nix build .#xcecore

You can find the generated binaries at =result= directory

构建现在是**完全 hermetic(离线)**的,无需再加 --option sandbox false。 所有 Go 依赖都已 vendored(go mod vendor,提交到各模块的 vendor/ 目录), Python 组件 xceadmin 也通过 pip install --no-build-isolation --no-index 从 nix 环境安装,构建阶段不再访问网络。因此在开启 sandbox(sandbox = true) 的机器上也能直接构建,不会再出现向 goproxy.cn / PyPI 请求而 DNS 解析失败的问题。

3.3.1 更新 Go 依赖(vendor 维护)

Go 模块(frontend/apiserver、frontend/xcecli、 autoscaler/providers/baidu/src/bcctool)的依赖以 vendor/ 目录的形式提交进仓库, 构建时 tools/go_build 使用 GOFLAGS=-mod=vendor、GOPROXY=off 完全离线编译。

当修改了某个模块的 go.mod / go.sum(新增、升级、删除依赖)后,需要在 nix develop 环境内重新生成对应模块的 vendor/ 并提交:

cd xce/src
nix develop

# 以 apiserver 为例,其他两个模块同理
cd frontend/apiserver
go mod tidy
go mod vendor
cd -

# vendor/ 必须被 git 跟踪,否则 flake 构建(只读取 git 已跟踪/暂存的文件)看不到它
git add frontend/apiserver/go.mod frontend/apiserver/go.sum frontend/apiserver/vendor

注意:nix build 从 git 树取源码,未被 git add 的 vendor/ 文件不会进入构建, 会导致 -mod=vendor 报 inconsistent vendoring 而失败。改完依赖记得 git add vendor/。

3.4 build (for daily develop)

日常开发推荐直接使用 src/tools/xce-build.sh(若在 Claude Code 中开发,对应的是 /xce-build skill), 它会执行 meson setup / meson compile,并在 asan profile 下跑测试。该脚本会自动进入 nix develop 环境(脚本开头检测 IN_NIX_SHELL,未设置时自动用 nix develop 重新执行自身),无需手动 nix develop 或额外包一层,直接跑即可:

cd xce/src

tools/xce-build.sh              # 全部组件 + 全部测试 (asan profile, 输出到 xce.build/)
tools/xce-build.sh controller   # 只构建/测试 controller (lib + controller + xce_test)
tools/xce-build.sh worker       # 只构建 worker
tools/xce-build.sh scheduler    # 只构建/测试 scheduler
tools/xce-build.sh apiserver    # 只构建/测试 apiserver + xcecli (Go)

(如果已经身处 nix develop 交互式 shell 内运行以上命令,脚本会检测到 IN_NIX_SHELL 已设置,跳过重新 进入的步骤,直接执行构建。)

支持三种 profile(tools/xce-build.sh [component] [profile]):

  • asan(默认):debug + AddressSanitizer,输出目录 xce.build/,会运行测试
  • debug:-O0 -g3 无 sanitizer,输出目录 build.dbg/,仅编译,用于 gdb 调试
  • release:-O3 无 sanitizer,输出目录 build.release/,仅编译,用于 soak/性能测试

注意:单组件构建默认会先 rm -rf 对应的 build 目录再重新配置,若同一目录下已有其他组件的产物会被清掉; 需要在同一目录下累积多个组件时加 --incremental(增量 reconfigure,不清空目录)。

也可以手动执行底层的 nix + meson 命令(xce-build.sh 实际上就是对下面步骤的封装):

cd xce/src
nix develop

# using meson to generated build configuration (debug and address sanitize)

meson setup xce.build --buildtype=debug -Db_sanitize=address

# run meson to build
meson compile -C xce.build

# The generated files will be in xce.build/

3.5 build AppImage package

cd xce/src
nix build .#xce.appImage

This will create an all-in-one xce.AppImage (available at ./result) which contains everything for package delivery.

4. misc

4.1. Windows remap ctrl and alt key

create a reg file, e.g. remapkey.reg

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,05,00,00,00,38,00,1d,00,1d,00,38,00,38,e0,1d,e0,1d,e0,38,e0,00,00,00,00

then double click remapkey.reg to import it. Restart OS.