admin-plane breakpoint 2: YAML resourcemap wiring + queue payload deep-copy
Breakpoint 2 of the admin-plane gaps (YAML flag/resourcemap wiring), plus
the queue payload ownership fix that the t1 controller segfault of
2026-10-23 12:41:13 traced to.
1. YAML resourcemap wiring (admin-plane-gaps 1.10):
- schema: Resource.distributions @8 / hostBased @9 uncommented --
resourcemap had no wire destination at all before this.
- resobj.c: resobj_init's #if 0 stub replaced with a real deep copy
(distributions + hbased_defined); resobj_destroy frees them. Deep
copy is mandatory: the source is myconf->cluster_config, which
keeps owning and later frees its tree.
- controller: reconcile_config_resources + seed_sharedinst_from_config
(cluster.c) seed c->sharedinst for SHARED|GLOBAL resources whose
distribution carries a value (the license-pool shape), from both
config_activate_committed paths, ahead of the conf4sched push.
- xceadmin: mapping.py maps resourcemap -> distributions (only the
location:[all]+value shape sets SHARED|GLOBAL), build.py renders
shared=true/distributions; golden expected.bin regenerated.
2. xcesched_push_configs resval double free (t1 12:39:52 abort): the
seeded push freed the resval entries AND the backing array via
free_name_val_list, then xlist_fini freed the same array again.
Detach xreslist before the FREEUP/xlist_fini block (same ownership
class bug-1506 fixed in sched_worker_change). Latent since inception:
every pre-1.10 push ran with nresvals==0 and never armed it.
3. queue payload deep-copy (t1 12:41:13 encode_Threshold_list segfault):
queue_init (create) MOVED the definition's users/hosts/thresholds/
window payloads out, and queue_refresh_def MOVED them out again --
but reconcile_config_queues runs MORE THAN ONCE over the same
cluster_config object (boot: pre-election activation AND phase2's
build_queues), so the second pass refreshed live queues from the
drained definition: thresholds became the poisoned (count>0,
array=NULL) pair, the generated encode_Threshold_list has no NULL
guard, and the next queue/worker push died on the NULL base.
Both paths now deep-copy (queue_init and queue_refresh_def); the
definition keeps its payloads, (NULL/0) genuinely means "no list in
this config", and reconcile is idempotent. New dup_group_config
helper in utils.c (the generated code has no GroupConfig dup).
Mirrors resobj_init's distributions deep-copy and worker.c
apply_join_state's dup pattern.
4. UT: test_reconcile_thresholds_survive_second_pass drives a triple
reconcile of a threshold-carrying queue and asserts definition
intactness plus a real encode_QueueConf -> wire -> decode round
trip; new test_config_resource_reconcile.c covers seed/deep-copy/
non-seed classes and idempotent re-seed.
Suites: controller_unit_test 1348/1348, sched_push 4, sched_worker_change 3,
apiserver_push 2, host_runusage 11, xcelib 759, xcealloc_oom 11,
99 scheduler binaries, all worker + harness suites, xceadmin pytest 375.
t1 E2E: boot double-reconcile clean, scheduler rejoin + full conf4sched
push applied, recovered crash-era PEND job reached DONE, 30-job saturation
flow (24 RUN + 6 PEND -> DONE), live gen-12 config activation with
re-push to the connected scheduler -- no crash, all four daemons healthy.
Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
parent
563f8e6890
commit
fc33a5da16
22 changed files with 1510 additions and 63 deletions
|
|
@ -197,5 +197,29 @@
|
|||
- 显示面终验:push priority 45→46→47(无 apiserver 重启)后 `xce queues list` 即时显示 47;作业 `xce jobs submit -q short` → DONE 全流程过,四 daemon 健康。
|
||||
- 运维注意:build.dbg 产物是 system-interpreter + nix-lib 混链(裸跑段错误);部署须用 `patchelf --set-interpreter /nix/.../glibc-2.42-47/lib/ld-linux-x86-64.so.2 --set-rpath <nix 库路径>` 后再 `--help` 预检。
|
||||
|
||||
### 断点 ②:YAML flag/resourcemap 接线(待做)
|
||||
### 断点 ②:YAML flag/resourcemap 接线(已实施;单测全绿 + t1 E2E 全通过)
|
||||
|
||||
**四处修复**(admin-plane-gaps 1.10):
|
||||
|
||||
1. **schema**(src/schema/controller.capnp):`Resource.distributions @8` / `hostBased @9` 取消注释(此前被注释掉——resourcemap 的目标字段根本不在 wire 上)。ResourceDistribution 结构本就活跃、编解码器早已生成,schema 改动后 encode/decode/free_Resource 自动带上 distributions(`decode_ResourceDistribution_list` 分配的正是 `res_distribution_t*`,与 C 结构零胶水)。
|
||||
|
||||
2. **resobj 深拷贝**(src/lib/resobj.c):`resobj_init` 的 `#if 0 /* FIXME */` 存根替换为真深拷贝——distributions 逐项深拷(value 按 which 拷贝、svalue strdup、hosts strdup),hbased_defined 深拷;`resobj_destroy` 对应释放。**必须深拷不能 MOVE_PTR**:源是 myconf->cluster_config(持续持有并在下次激活时 free),MOVE_PTR 会把配置树偷空。新增 `resval_deep_copy`/`resval_free` 静态助手(与 update_val_res/resval_obj_killer 同一所有权约定)。调度器 makeResource/resourceApplyDef 路径同文件同修复(推送上来的定义也能带 distributions)。
|
||||
|
||||
3. **控制器 reconcile**(src/controller/cluster.c + config_repl.c):新增 `reconcile_config_resources(c)`——`cluster_build_resources`(boot 建条目、激活重跑 EEXIST 容忍)+ `seed_sharedinst_from_config`(对 SHARED|GLOBAL 且 distributions[0] 带 value 的资源,deep-copy 该 value 进 `c->sharedinst[live->idx]`,zarray_put 换值语义=幂等激活直接热更新容量)。**seed 用 resmgr 的运行时 idx(key_add 分配),不能用 config 自带的 idx 字段(YAML 来源恒 0)**。调用点三处:boot hook(xce_config_init)、config_activate_committed 的 -ENOENT fallback 路径与 hash-verified 路径(均在 `xcesched_push_configs` 之前,保证下一次 conf4sched push 即携带 seed)。调度器零改动——core_apply_config_resvals/sched_init_sharedInst 本就把 conf4sched.resvals 送进 core->sharedInst,calc_wanted_global_resvals 以 XCE_RESF_GLOBAL 门控 job wants。
|
||||
- **YAML 映射规则**(mapping.py):只有 `location: [all]` + `value` 形态置 `SHARED|GLOBAL`(0x08|0x04,即 license 池场景);其余形态(host 子集/多段/per-host)忠实映射进 distributions 但不置位——host 子集静态共享池需要 elastic-shared-resources 调度器工作,不在本次范围。
|
||||
|
||||
4. **Python 工具链**(xceadmin):mapping.py 新增 `RESOURCE_FLAG_SHARED/_GLOBAL/_SHARED_GLOBAL` 常量 + `_build_resource_value`(YAML 值按类型选 union 成员:bool→bvalue、number→fvalue、string→svalue,经 `.u` 组访问)+ `_build_resource` 映射 resourcemap→distributions;build.py 移除 `_UNMAPPED_BUT_VALID_ITEM_KEYS["resources"]` 的 resourcemap 条目、`_format_resource` 增加 `shared=true` 与 `distributions=[hosts]=value` 渲染(`(reported)` 表示动态无 value 形态)。schema 单点编辑即可——meson-build.sh 构建时把 src/schema/controller.capnp 同步进 xcecommon/schema_files/。
|
||||
|
||||
**单测**(全绿):
|
||||
- Python:`test_mapping.py` 新增 `test_map_list_section_resourcemap_maps_distributions_and_all_flag`(四形态逐一断言)+ `test_map_list_section_resourcemap_real_repo_configs`(真 configs/resources.yaml 全量分布数核对);`test_config_build.py` 的过时 `test_build_warns_not_fails_on_schema_valid_unmapped_resourcemap_field` 替换为 `test_build_maps_resourcemap_to_distributions`;full golden `expected.bin` 再生(5312→6696 字节,distributions 上 wire 的直接证据;minimal fixture 无 resources.yaml 不受影响)。373 用例全绿。
|
||||
- C:`test_config_resource_reconcile.c` 两个用例——注册+seed+深拷非别名断言、SHARED-only/无 value/非 shared 三类不 seed 断言、幂等重跑+容量热更新(100→250)断言;controller 套件 1347 tests 全绿,ASAN 套件 6/6 OK。
|
||||
|
||||
**语义注意**:`resourceApplyDef`(调度器)只比较 valtype/ordtype/flags/interval——已注册资源定义变更(含 distributions)报 RES_APPLY_CHANGED 但不 apply(重启拾取);而控制器侧 sharedinst seed 走 value 换入,容量热更新。分布结构变更(新增 host 段)目前不热更新到已注册调度器对象,与既有"定义变更需重启"语义一致。
|
||||
|
||||
**E2E 结果(2026-10-23 t1,14:20–14:28)**:修复后 controller 部署重启 → boot 双 reconcile 正常(14:20:30 pre-election 激活建 4 队列 + phase2 xce_config_init 二次 reconcile,旧代码在此中毒)→ scheduler 重连,conf4sched push 完整应用 → 作业流全通过:恢复的 12:41 卡死 PEND 作业走到 DONE;rusage[staticres4=1] 作业 PEND→RUN→DONE;30 作业饱和压测(24 槽全占 + 6 PEND,槽释放后全部收敛 DONE);live config push(gen 12 激活,4 队列 "updated by configuration activation" + re-push 到已连接 scheduler)**不再崩溃**——这正是 12:41:13 旧代码 segfault 的同形场景。全部 4 daemon 健康,t1/t2/t3 OK。
|
||||
|
||||
**修复中发现并修掉的深层缺陷(queue 载荷所有权)**:bp2 最初只修了 sched.c 的 double-free;t1 上 12:41:13 的 encode_Threshold_list segfault 根因更深——**queue 的 capnp 载荷(users/hosts/thresholds/windows)在创建路径(qobj.c queue_init)被 MOVE_PTR 消耗掉定义**,而 reconcile_config_queues 对同一 cluster_config 对象会跑多次(boot:pre-election 激活一次 + phase2 build_queues 又一次),第二次 refresh 读到已掏空的定义:thresholds 是 count>0/array=NULL 的毒对(MOVE_PTR 置空指针但不落 count),generate 的 encode_Threshold_list 无 NULL 卫兵,下次队列/worker push 即崩。修复=创建与刷新两条路径都改**深拷贝**(定义归 cluster_config 持有并在激活换装时 free;镜像 resobj_init distributions 深拷与 worker.c apply_join_state 的 dup 模式):qobj.c queue_init 深拷 users/hosts/thresholds/windows_config;queue.c queue_refresh_def 深拷同四项(定义里 NULL/0 从此真表示"配置无此列表",刷新清空运行时队列是正确语义);新增 `dup_group_config`(src/lib/utils.c,生成代码无此 dup)。UT:test_reconcile_thresholds_survive_second_pass 驱动三连 reconcile + 真 encode_QueueConf→wire→decode 回读断言(1348/1348 绿;全套件扫描 sched 99 个/worker/harness/xcelib/xceadmin 375 全绿,sched_repl_trace_baseline_test 一个既有失败经 stash 验证与本改动无关)。
|
||||
|
||||
**运维观察(未修,记录)**:CLI `config push` 单节点 ackedCount==0 误报拒绝(push_config_to_standbys 只数 standby 传输层 ACK,controller 端实际成功——以 controller 日志为准);scheduler 对已注册资源定义变更报 WARN 不 apply(重启拾取——本次 scheduler 重启后 staticres4/res1 正常注册,作业可调度);饱和时 PEND reason 显示 `-`(PEND-reason 发布链路对 NO_RESOURCE 场景仍未出端到端值,P1 遗留);t1 controller 无 core dump(ulimit 0)——12:41 segfault 现场只能靠 objdump 模式匹配定位。
|
||||
|
||||
### 断点 ③:done 作业 live GC(待做)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue