diff --git a/admin-plane-gaps.md b/admin-plane-gaps.md new file mode 100644 index 00000000..6ac12495 --- /dev/null +++ b/admin-plane-gaps.md @@ -0,0 +1,201 @@ +# XCE 管理面能力评估（查漏补缺 · 第一步） + +> 生成时间：2026-10-23。评估方法：源码逐模块审计（CLI/REST/controller/scheduler/worker/configs 四路并行）+ t1/t2/t3 真集群 CLI 探针验证。 +> 范围：按用户 10 节检查清单（1.1 管理cluster … 1.10 shared resource）逐项对照。 +> 约定：✓=具备，△=部分具备/有缺口，✗=缺失。调度/性能类问题不在此重复，见 gaps.md 对应条目。 +> 本文档只做记录，未做任何代码改动。 +> +> **2026-10-23 更新**：断点 ①（配置可达性）已实施，见文末《断点修复记录》。 + +--- + +## 1. 管理 cluster + +### 1.1 使用 cluster + +| 子项 | 判定 | 现状与证据 | +|---|---|---| +| 查询 cluster 信息 | △ | `xce id` → GET /xce/v1/clusters（routers.go:118，controllers/clusters.go:18）。**但实测输出是空壳**：`My cluster name: <>`、`The cluster status: <>`。ClusterInfo.cluster_admins 注释明言"never populated by config push"（xcecli backend/handler.go:225）。集群名实际存在（xce.yaml cluster_name: eda1），没接到这个端点。 | +| 配置管理员 | △ | RBAC 完整：`roles add/delete/assignpermission`、`users assignrole/unassignrole`、`permissions list`（23 权限、6 角色，rbac.yaml + includes 继承）。缺口：① **用户无 CRUD**——CLI/REST 都没有 add/delete user、改密码端点，加用户只能改 auth.yaml + 重启 apiserver；② 角色/用户角色是内存 DataStore，**推断不随重启存活**（API key 已持久化 state/apikeys.json；.kiro/specs/rbac-persistence 目录已规划未实施）；③ 引导方式是 auth.yaml 内置 dev 默认 admin/admin（有告警日志）。 | +| 控制 daemons | ✗ | 见 1.2。 | +| 使用命令配置 cluster | △ | 有整包配置通道：`xceadmin config build/push/dump`（YAML→cluster.bin→controller，SHA-256 校验、WAL majority barrier、`--dry-run` 真预检 bug-1506）。**没有对象级运行时命令**（无 `queue create/modify`、`resource add` 之类），改配置=改 YAML→build→push。热生效矩阵见 1.5 与下表。 | + +**配置热生效矩阵**（关键结论）： + +| 变更域 | 热生效？ | 证据 | +|---|---|---| +| 新增队列 | ✓ | reconcile_config_queues 建缺失队列（queue.c:51） | +| 修改既有队列（priority/limits/window） | ✗→✓ | 队列 reconciliation 曾明确"existing runtime queue: left untouched"（queue.c:51）；**断点 ① 已修**：queue_refresh_def 原地刷新 CONFIG-owned 字段（见文末修复记录） | +| 队列/主机 open/close（运行态） | ✓ | queue_ctrl/worker_ctrl + WAL 回放 restore_queue_state | +| controller 心跳参数 | ✓ | cluster.c:2238 活读 heartbeat_interval×count | +| worker 参数 | △→部分 | 仅 worker_resync_window_ms 且仅 join 时（controller_connector.c:1852）；confvars（heartbeat_interval/inactive_heartbeat_count）**断点 ① 已修**：激活时刷新 configmgr 条目并随 push 下发 | +| **scheduler 的 queues/limits/groups/resources** | ✗→✓ | `xcesched_push_configs` 曾唯一调用点是 on_scheduler_join（sched.c:139/774）；**断点 ① 已修**：config_activate_committed 激活缝 re-push 到已连接 scheduler（见文末修复记录） | +| apiserver | △ | 仅 join 时全量推（apiserver.c:648） | + +### 1.2 管理 daemons + +**✗ 整体缺失（已知且已规划）。** + +- 无任何 CLI/REST daemon 生命周期命令；deployment.md:287 原话："No process supervision is shipped. No systemd unit, no Dockerfile, no docker-compose, no supervisord config exists anywhere in this repo for any of the four daemons." +- 存在但未暴露：`OPCode_ctrlstatus`（HA 身份/健康快照：primary/responder/peer 状态 ok/catching-up/dead，capnp:556-574）只在 C client 库（client.c:420 xcecli_ctrlstatus），Go apiserver/xcecli 均未接。 +- `xceadmin scheduler record start/stop/status` 是唯一"控制运行中组件"的面（事件录制，非 daemon 生命周期）。 +- .kiro/specs/xce-node-manager 有完整 requirements+design（每节点监督 daemon、按角色启停组件、UDS 心跳、controller primary/standby 驱动 scheduler 启停）——**规划与缺口精确对齐，未实施**。 + +**缺口清单**：① daemon 状态/健康查询命令（哪怕只读）；② HA 状态查询与 standby 管理（现在只能看 daemon 日志）；③ 进程监督层（xce-node-manager spec 落地）。 + +--- + +## 1.3 管理 hosts + +| 子项 | 判定 | 现状与证据 | +|---|---|---| +| hosts 状态 | ✓ | 位标志：UNAVAIL/LOCKEDU/LOCKEDW/BUSY/CLOSED/HIDDEN/CLOSED_BY_ADMIN/RECONCILING（types.h:170-186）；显示优先级 UNAVAIL>CLOSED_BY_ADMIN>…>OK。实测 `hosts close t2` → CLOSED_BY_ADMIN 正确显示。 | +| 查询 hosts 信息 | ✓ | `hosts list`（-a/-l/-o/-w/--json/-t）+ `hosts select -R`（resreq 表达式选主机 dry-run）。小缺口：`-o` 仅 8 字段（hosts.go:121-130），-l 里的 CPUF/负载阈值/dispatch window 不进 -o；`-a/--host_info` 与 `-l/--verbose` 绑同一变量（list.go:189,193）。 | +| hosts 控制 | ✓ | `hosts open/close [--all] [-c comment]`（worker.c:985-1046，WAL 持久化）。实测 close→作业不受影响继续跑、控制命令仍可达（bug-1941 语义）。死枚举：HOST_TAG/HOST_UNTAG 无 C handler（types.h:629-635，默认 -1）；无重启主机命令。 | +| 增加 host 到集群 | △ | 路径=hosts.yaml 加条目→config build/push→worker join（join 必须 hostconfig 匹配否则 -EINVAL，bug-1494）。join 后 worker 以 upsert 增量到达 scheduler（XCE_SCHED_OP_UPDATE_WORKERS→core_upsert_worker）→ **新主机可 live 可调度**。缺口：validate_host 是 stub（join.c:48）；新主机上的 limit/group 类配置受"push 不达 live scheduler"（gaps #12）影响。无 `host add` 命令，纯文件流。 | +| 从集群删除 host | **✗** | **无任何删除 API/命令**。XCE_WORKER_HIDDEN（"Admin explicitly removed"）定义了但**全库无 setter**；worker 对象不死（resync.c:253：no code path ever key_del()s；worker_obj_killer 是空函数 cluster.c:1598）。从 hosts.yaml 删掉主机+push：运行中 worker 对象原样保留；其上作业被 re-home 到合成 lost_and_found 主机（config_repl.c:1173-1217，无 worker 端点=永远 PEND）。GC 管道（HIDDEN/UNAVAIL TTL → checkpoint 丢弃）存在但无入口。.kiro/specs/elastic-hosts 已规划未实施。 | +| host groups | △ | 定义 ✓（hosts.yaml hostgroups，members/exclude_members）、装载 ✓（core->hgmgr，snapshot 也载，bug-1943）、**消费仅 limit 域**（xcelimit.c:292-341）。**不能用于 select[]**（xce_select.peg 只解析资源名，无组语法）——LSF 式 `select[hgroup]` 放置能力没有。RBAC 有 hostgroup:view/control 权限但零管理 API；且 hostgroup 域的 host:control **未强制**（hosts.go:495 TODO，integration_test.go:556-593 记录为越权 bypass）→ 安全缺口。 | +| host 级作业异常处理 | △ | 自动侧齐全：worker 死→RUN 变 UNKWN\|TRANSIENT（bug-1509）由 rejoin/resync 解决；确认丢失→EXIT(-1 "worker lost")；派发失败→rePEND（bug-1494 系）；rejoin→RECONCILING 圆（bug-1963）。缺口：① 90 秒检测窗（gaps #9）；② 无管理员侧工具——没有"把主机 X 上的作业重排/requeue"命令、没有主机故障恢复策略配置（.kiro/specs/host-failure-job-recovery 已规划）；③ UNAVAIL 主机 resInUse 冻结是有意设计但无文档。 | + +--- + +## 1.4 管理作业 + +| 子项 | 判定 | 现状与证据 | +|---|---|---| +| 作业状态 pend/run/done/exit | ✓ | 全状态机：PEND/RUN/DONE/EXIT/PSUSP/USUSP/SSUSP/PDONE/PERR/WAIT/TRANSIENT/UNKWN/PROV/ERR（types.h:382-405）；迁移强制单调（apply_job_update_transition job.c:3519）；终态集 EXIT\|DONE\|PERR；PERR 合并 post-exec 失败（job.c:3376）。 | +| 查询作业信息 | ✓ | `jobs list` 过滤器丰富（-a/-u/-q/-p/-r/-d/-s/-J/-Jd/-P/--sla/-A），31 个 -o 字段，--json。**缺口：exit code 不上 job API**——C 侧已捕获（JobExitInfo.exitcode，bug-1867）、SSE job.finished 事件带 exit_code（broker.go:41），但 REST 行的 ExitStatus 字段无 writer（convert_pure.go:646 不映射，backend/ 零引用），CLI 恒 null；只能从 DONE/EXIT 位或 SSE 推断。已记 memory（prepost-exec 测试）。 | +| 作业控制 kill/signal | ✓ | kill（SIGINT→SIGTERM→SIGKILL 优雅链）/stop（SIGTSTP→SIGSTOP）/resume（SIGCONT）/checkpoint/restart（按原 spec 重提新作业，要求原提交者）/peek（实时输出流）；`kill -s ` 发任意一次信号（31 信号表），`--force` 即刻 SIGKILL；controller 对任意 signo 通用处理（exit code=128+signo，job.c:2207-2243）。缺口：job_ctrl_type_t 里的 mig/requeue/modify/switch/top/bot 在 REST 400 不可达（controllers/jobs.go:2187）——**mig（迁移）与 requeue 是 LSF 核心管理动作，此为最值得补的控制面**。 | +| job groups | ✗ | 装饰性字段：JobGroup 在 Go model/控制过滤器里，但不在 wire JobSubmitData（capnp:583-595），modelJobToSubmit 不拷贝，无生命周期管理无消费者。作业依赖（DependCond）同样：纯字符串字段，调度器/controller 零解析零求值。数组作业 ✓（.kiro/specs/job-dependency、gang-groups 已规划未实施）。 | +| 作业异常处理 | △ | PEND reason 全链路存在：9 个命名码（preason.h）scheduler→controller→JSONB→CLI。缺口见 gaps #6（alloc_rc<0 无 reason）、#7（ALTER 子作业 reason 只取一个）、#15（策略性拒绝显示 '-'）。**实测再证**：主机 CLOSED_BY_ADMIN + select(hname==t2) 作业 PEND，`pending_reason` 显示 '-'。 | +| done 作业清理 | **△** | 配置存在：finished_retain_s=3600（capnp PersistConfig.finishedRetainS）。执行点唯一：collect_job_frames 只把过期作业排除在**新 snapshot**外（persist.c:2558/8838）。**live 表无 GC**：设计文档里的 kvtab_gc 未实现（kvmgr.c:279/286 del_key/vclear=NULL；persist.c:7085 恢复期 retention pass 是 no-op）；无 purge CLI/API（job 无 DELETE 路由）；无归档库。**实测证实**：集群里 done 作业已积压一天+（Oct 22 03:10 的作业仍在 jobs list -d），长运行 controller 内存里只增不减，跨重启才消失。无回收站/历史查询。.kiro/specs/controller-job-cleanup 已规划未实施。 | + +--- + +## 1.5 管理 queue + +| 子项 | 判定 | 现状与证据 | +|---|---|---| +| queue 状态 | ✓ | 位标志 OPEN(0)/CLOSED/INACT_WINDOW/INACT_ADMIN/NOPERM/DISC/FAIRSHARE_DEFAULT/HIDDEN（types.h:413-422）。语义正确且对齐 LSF：**closed 只挡新提交、PEND 作业照常派发**（sched_queue_is_ready 只查 INACT 位，core.c:685-715）；inactive 挡派发（INACT_ADMIN→PREASON_QUEUE_DRAINING，INACT_WINDOW→TIME_WINDOW，scheduler.c:499-520）；closed 提交被拒 -E_XCE_QUEUE_CLOSED（job.c:2616-2721）。实测两组语义均符合。 | +| 查询 queue 信息 | △ | `queues list`（-u/-q/-o/-w/-l/--json）。**新发现显示缺陷：closed 期间 `queues list` 仍显示 `open:active`**（实测两次：close night 后提交被拒"queue is closed"，同时 list 显示 open:active；重新 open 后提交成功）。原因指向 QUEUE_STAT_OPEN==0x00 是"无位"默认渲染、CLOSED 位没有对应的显示分支，或 queue 控制变更未推送 apiserver 内存库——需定位。另外 sched_delay/accept_interval 是死配置（gaps #4，本次独立复核确认：schema 就没有这两个字段，capnp 有位无源）。 | +| queue 控制 | ✓ | `queues open/close/active/deactive [--all] [-c]`（queue.c:128-157，WAL 回放 restore_queue_state），四个动作全通 REST。实测 close/open/active 均即时生效。注意动词拼写是 active/deactive（非 activate）。 | + +--- + +## 1.6 host-based resource + +| 子项 | 判定 | 现状与证据 | +|---|---|---| +| 概念/定义/配置 | ✓ | resources.yaml 全谱系：内置负载指数、host 属性（type/model/cpuf/maxmem/…）、static（shared/hostbased）、dynamic、boolean/string、resourcemap；JSON Schema 校验；join 时按 hostconfig 播种（mark_host_res/add_hostattrs_resvals/set_host_slots，worker.c:232-341）。 | +| resource 种类 | △ | valtype STRING/NUMERIC/BOOLEAN、ordtype INCR/DECR/NA、9 个 flags 齐备（types.h:81-89）。**关键接线断裂：YAML→capnp 映射只映射 CONSUMABLE 和 RELEASE 两个 flag**（mapping.py:194-200）——DYNAMIC/SHARED/GLOBAL/EXTERNAL/BUILTIN 从 YAML 永远置不上；resourcemap 在 build 时被静默丢弃（build.py:38-65 _UNMAPPED_BUT_VALID_ITEM_KEYS，capnp:192-206 目标字段被注释）。ordtype 存了但**无消费者**（排序纯 first-fit，关联 t1 亲和）。 | +| load index | ✓ | **11 个内置指数全部真采集**：r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io，~15s 周期（resource_sync.c:60-106），实现实在（node_information.c：/proc/loadavg、utmp、MemAvailable、statvfs、diskstats…），推送链完整（resvals→controller opupd.c→RESVALS_DELTA→scheduler core_events.c:1228-1263）。实测 distribution list 有 io/it/ls/mem 值。 | +| static resource | △ | 声明与播种 ✓（hostbased 静态值随 join 落地）；但 share 型静态资源的 resourcemap（实例/份额定义）在 build 时被丢（见上），**静态 share 资源实际不工作**。 | + +--- + +## 1.7 外部 resource（external resource） + +**✗ 缺失。** + +- XCE_RESF_EXTERNAL(0x10) 定义了但全库无 setter。 +- 无 elim 式外挂采集器：worker 没有子进程/脚本采集路径；`resource_sync_register_load_index` 是进程内 C API（给 xcecd 内部代码用），无任何调用方。 +- dynres1/dynres2 声明了 interval 但**无采集者**——定义在表里、值永远不会自动出现，只能管理员手工 update_resval 推。 +- scaler.yaml（aws-asg/k8s-pool）映射为 hproviders 推给 scheduler，但 **scheduler 零消费者**（而且它是自动扩缩 provider，不是资源采集器）。 +- .kiro/specs/custom-resource-reporting 已规划未实施。 + +--- + +## 1.8 user groups + +| 子项 | 判定 | 现状与证据 | +|---|---|---| +| 定义 | ✓ | users.yaml usergroups（ug1/ug2）→ config push → apiserver store（clib.go:523）+ scheduler core->ugmgr（core.c:675）；capnp GroupConfig{name,members,exmembers}。 | +| 查询 | **✗** | **无任何查询通道**：REST 无 usergroups 端点、CLI 无命令（RBAC 有 usergroup:view 权限但没地方用）。管理员配了 ug1/ug2 后，除了翻 users.yaml/cluster.bin dump，无法从系统里看到组与成员。 | +| 使用 | △ | 两个消费者：① self_ug 作业可见域（同组成员互见作业，job_scope.go:40，bug-1934）；② **limit 域**——limits.yaml 按 user group 定 per-user/per-user+queue 上限（limit_config.c:139 build_limit(ugmgr,…)）。**不用于 fairshare**（无 fairshare，gaps #3）。另发现：queues.yaml 的 users/hosts 成员表（队列级成员限定）**在调度器无消费者**——配置 carried but not enforced，属新发现的死配置。 | + +--- + +## 1.9 user mapping（用户映射） + +**✗ 缺失。** + +- 无 submit-as/runs-as 映射实体（无 UserMap 配置、无 exec_user 提交字段）。 +- 执行用户决策链：REST 无 exec_user → apiserver `resolveExecUser` 取 submit user（convert_pure.go:360-370，bug-1780）→ JobStartInfo.execuser（capnp:317）→ worker 按 /etc/passwd 存在性执行，**不存在则静默 fallback 到 nobody**（controller_connector.c:1061-1083）。 +- model 的 exec_uid 字段恒 0（job_information.c:27）不填。 +- 关联风险：user mapping 缺失 + nobody fallback 静默，是 gaps #17（执行上下文无显式契约）在账号维度的表现。 +- 另注意：RBAC 用户（auth.yaml，apiserver 域）与集群 usergroups（users.yaml，config 域）是**两个独立用户平面**，互不引用——管理员视角没有统一的"用户"实体。 + +--- + +## 1.10 shared resource + +**△ 机制存在，端到端不通。** + +- 机制 ✓：SHARED flag 资源进集群级共享池 c->sharedinst（worker.c:65-143 按 flag 路由）；调度两段式——先集群级 precheck（sched_alloc_prepare_shared_，alloc_runner.c:73-132，不满足→无主机 PEND 行+PREASON_NO_RESOURCE/LIMIT_GLOBAL）再逐主机检查；snapshot 以 host==NULL 池形态携带恢复（bug-1923）。limit vres（GLOBAL limit）是当前唯一活的生产者。 +- 断点 ✗：YAML 映射不置 SHARED flag（mapping.py:194-200）+ resourcemap（LSF 式静态份额/实例定义，如 dynres2 host1:50/host2:50）build 时丢弃 → **静态共享资源（license 类场景）声明得、用不上**；实测 distribution list 只有 hostbased 行，无共享实例行。 +- .kiro/specs/elastic-shared-resources、resource-instances 已规划未实施。 + +--- + +## 二、本次新发现缺陷清单（不在 gaps.md 的） + +| # | 缺陷 | 证据 | +|---|---|---| +| N1 | `xce id` 空壳：集群名/状态/管理员全空 | 实测输出 `<>`；handler.go:225 注释 | +| N2 | `queues list` 显示与准入不一致：closed 队列显示 open:active | 实测（close night 后提交被拒、list 仍 open:active） | +| N3 | exit code 不上 job API（REST ExitStatus 死字段） | convert_pure.go:646 无映射；SSE 有 exit_code | +| N4 | done 作业 live 无 GC、无 purge API、无归档：长运行 controller 终态作业内存只增不减 | persist.c:2558 唯一执行点；kvmgr.c:279 del_key=NULL；实测积压一天+ | +| N5 | `jobs peek` 对已结束作业 501 | 实测 GET /jobs/:id/output status 501 | +| N6 | `jobs submit --configfile` 不解析配置行，拼进命令行当作业命令执行 | 实测：job.xce 两行变成 command "queue=short job_name=cfgtest2" | +| N7 | 用户无 CRUD：无 add/delete user、无改密码端点，加人=改 auth.yaml+重启 | routers.go 全量；users 命令族仅角色指派 | +| N8 | hostgroup 域 host:control RBAC 未强制（越权 bypass） | hosts.go:495 TODO；integration_test.go:556-593 | +| N9 | user groups / host groups 无查询 API/CLI（usergroup:view 权限无处可用） | routers.go 无端点；CLI 无命令 | +| N10 | queue users/hosts 成员限定配置无消费者 | 队列 GroupConfig 载入但调度器无检查 | +| N11 | HOST_TAG/HOST_UNTAG 枚举无 C handler（死枚举） | types.h:629-635；worker.c 默认 -1 | +| N12 | ordtype（INCR/DECR 排序意向）无消费者：放置纯 first-fit | 关联 t1 亲和（memory xce-firstfit-t1-affinity） | +| N13 | CLI 小毛病：`id` help 写着 `xce clusters`；hosts `-a/-l` 同变量；jobs list `-W` 死标志 | cmd/clusters/list.go:26；hosts/list.go:189,193；jobs/list.go:365 | +| N14 | 一次观察：closed 队列提交 CLI 挂死 >2min（重试快速失败，未复现，需确认） | 本轮探针 02:43 现场 | +| N15 | worker 执行用户不存在时静默 fallback nobody | controller_connector.c:1061-1083（与 #17 关联） | + +## 三、与既有记录的关系 + +- gaps.md 一~七节（调度/架构/构建）全部继续有效，本文不重复；交叉引用已标注（#3 fairshare、#4 sched_delay、#6/#7/#15 PEND reason、#9 心跳窗、#12 配置传播、#17 执行上下文）。 +- .kiro/specs 的空目录（elastic-hosts、controller-job-cleanup、job-dependency、gang-groups、resource-instances、custom-resource-reporting、host-failure-job-recovery、elastic-shared-resources、rbac-persistence、project-* 等）正是本评估所列缺口的"已规划未实施"标记；xce-node-manager 有完整 spec。规划面与管理面缺口高度吻合，说明缺口识别一致，缺的是实施。 + +## 四、若按用户价值排优先级（供第二步讨论） + +1. **配置可达性**：让 config push 真正到达 live scheduler（gaps #12 的一半收益）+ 既有队列编辑热生效——"配了不生效"是管理面最大的信用问题。**→ 已作为断点 ① 修复，见下。** +2. **可见性补齐**（低风险高价值）：queues list 状态显示修复（N2）、exit code 上 API（N3）、cluster info 填实（N1）、user/host group 查询端点（N9）、daemon/HA 状态只读命令（ctrlstatus 已有协议只差暴露）。 +3. **生命周期补齐**：done 作业 GC+purge 命令（N4）、host 删除流程（elastic-hosts spec 落地）、用户 CRUD（N7）。 +4. **管理动作补齐**：requeue/mig 控制动作（枚举已在、REST 400 挡着）、host 故障作业恢复策略（host-failure-job-recovery spec）。 +5. **资源接线**：shared/external resource 的 YAML→capnp 断点（mapping + resourcemap 注释字段）、dynres 采集钩子（custom-resource-reporting spec）。 + +--- + +## 五、断点修复记录（2026-10-23） + +### 断点 ①：配置可达性（已实施；单测全绿 + t1/t2/t3 真集群 E2E 验证通过） + +**三处修复**（admin-plane-gaps 1.1）： + +1. **既有队列原地刷新**（src/controller/queue.c）：新增 `queue_refresh_def()`——`reconcile_config_queues()` 遇到已存在队列时不再跳过，而是在 `RTTI_LOCK_WRITE` 下原地刷新 CONFIG-owned 字段：resreq/pre_exec/post_exec（interned xstring 换引用）、users/hosts/thresholds/runwindow/dispatchwindow（free_* 旧值 + MOVE_PTR 接管新定义的 capnp 载荷，与 qobj.c queue_init 同一所有权约定）、attrs/status/priority/accept_interval/sched_delay（标量直拷——config 是 status 的 source of truth，WAL 回放的 queue_ctrl 事实仍按回放序生效）。刷新成功的队列走与新建队列相同的 `persist_notify_queue_updated_no_wal` NO-WAL 通知（t_apply_seam_notifying 守卫，不产生 WAL_QUEUE_CTRL 重复条目）。**约定变更**：返回值仍只计新建数，刷新数走 TRACE。 + +2. **confvars 激活刷新**（src/controller/cluster.c）：`cluster_build_config()` 改为幂等——boot 建条目（add_config），激活重跑时经新增 `config_refresh()` key_find 原地换 interned value 串（makeConfig 拒 EEXIST、kvmgr 无 del_key，只能原地换）。collect_confvars 导出的 heartbeat_interval/inactive_heartbeat_count 不再冻结在 boot 值。 + +3. **激活缝 re-push**（src/controller/config_repl.c 两处：hash-verified 激活路径 + -ENOENT fallback 路径）：`config_activate_committed()` 激活成功后依次 `cluster_build_config`（confvars 刷新）→ `mylock_lock(&c->sched.lock)` → `xcesched_push_configs(c)`（sched.c 去 static，cluster.h 出声明）。锁序安全：该缝已建立 apply_pump_lock → sched.lock 序（WAL_QUEUE_CTRL case 的 hook 触发 sched_queue_update 同序），无反向边（持 sched.lock 者从不取 apply_pump_lock；push 只入 endpoint outq）。推送失败仅 WARN——scheduler 保有旧配置直到下次 join/push。 + +**单测**（全绿，controller 套件 1345 tests）： +- `test_config_queue_reconcile.c`：新增 `test_reconcile_refreshes_existing_queue_in_place`（同对象指针、status/priority/resreq/pre_exec 原地生效、notify 恰一次）；原 bug-1469 测试更新计数语义（建+刷各一次 notify）。 +- `test_config_repl.c`：新增捕获 fake `xcesched_push_configs`（controller_unit_test 不链 sched.c）+ 两个测试：`test_config_activation_repushes_configs_to_connected_scheduler`（sched.ep 置零值 fd=-1 endpoint 时激活恰推一次、二次激活再推；marker commit 自身经 apply pump 触发的激活在 ep=NULL 时不推）与 `test_config_activation_deferred_does_not_push_configs`（-EAGAIN 未激活不推）。 + +**已知未修（记录）**：`xcesched_push_configs` 的 pre-existing 引用泄漏——clist/rlist/xreslist 只锁不放（unlock_collect_queues 仅处理 qlist），每次 push 引用计数累积。与本次改动无关，留待单独清理。 + +**真集群 E2E 验证（2026-10-23，t1/t2/t3，新 controller 已部署）**： +- 部署：新 controller 进 AppImage store closure（`/nix/store/h7i88.../bin/xce_controller`，md5 6121898），补 `subprojects/tinylog → /opt/xce-denix/lib` 软链（RUNPATH 需要且 libtlog 非 ASAN，md5 cab46c49）。 +- 激活链路：`xceadmin config push`（单节点 `ackedCount==0` 是已知误报，实际以 `Config committed on majority` 为准）→ gen=8→9 → `config activated from committed marker` → `queue xce/normal/night/short updated by configuration activation`（4 条全刷）→ `re-pushed configs to the connected scheduler` → scheduler 侧 `sched_init_resources` 在轮界触发（sched_apply_pending_config → sched_init_config 原地应用，bug-1886 路径）。第二次 push gen=9→10 复证。 +- **附带修复（发现于 E2E）**：apiserver Go 侧 `handleNotifyData`（backend/clib.go）缺 `NotifyData_data_schedUpdateQueue` case——controller 的 per-queue 通知被静默丢弃，`store.Queues` 冻结在 join 快照，`xce queues list` 看不到热应用的优先级/队列状态变化。新增 case → `OnQueueUpdated(cQueueToModel(qp))`；单测 `TestOnQueueUpdated_InPlaceRefresh`。部署新 apiserver 到 /opt/xce-denix/bin（旧版备份 xce_apiserver.bak.20261023）。 +- 显示面终验：push priority 45→46→47（无 apiserver 重启）后 `xce queues list` 即时显示 47；作业 `xce jobs submit -q short` → DONE 全流程过，四 daemon 健康。 +- 运维注意：build.dbg 产物是 system-interpreter + nix-lib 混链（裸跑段错误）；部署须用 `patchelf --set-interpreter /nix/.../glibc-2.42-47/lib/ld-linux-x86-64.so.2 --set-rpath ` 后再 `--help` 预检。 + +### 断点 ②：YAML flag/resourcemap 接线（待做） +### 断点 ③：done 作业 live GC（待做） diff --git a/src/controller/cluster.c b/src/controller/cluster.c index dde2d0ac..bc096d4e 100644 --- a/src/controller/cluster.c +++ b/src/controller/cluster.c @@ -1463,6 +1463,59 @@ static void endpoint_killer(void* p) return; } +/* config_refresh: update an EXISTING configmgr entry's value in place. + * makeConfig refuses EEXIST (configobj.c), so re-running cluster_build_config + * after boot could never refresh the confvars the scheduler/worker config + * pushes export -- they would stay frozen at their boot-time values even + * after a config activation changed them (admin-plane-gaps 1.1: config push + * never carrying the activated values). key_find the entry under the + * configmgr WRITE lock and swap the interned value string (dereference old, + * makeString new -- xce_config_t.value is an interned xstring, same + * ownership convention qobj.c/jobj.c use for resreq). + * + * Caller must NOT hold c->configmgr. Returns 0 on refresh (or "entry absent + * -- nothing to refresh"), negative on a real error. */ +static int config_refresh(cluster_t* c, const char* name, const char* value) +{ + xce_config_t* cv = NULL; + int rc; + + if (c->configmgr == NULL) { + /* No config manager (bare/test fixture) -- nothing to refresh; + * add_config's caller owns creation on that path too. */ + return 0; + } + + lock(c->configmgr, RTTI_LOCK_WRITE); + rc = key_find(c->configmgr, (void*)name, (void**)&cv); + if (rc > 0 && cv != NULL) { + /* bug-447 precedent: binary key_find returns 1 on found. */ + char* nv = makeString((char*)value); + if (nv == NULL) { + unlock(c->configmgr); + return -ENOMEM; + } + if (cv->value != NULL) { + dereference(cv->value); + } + cv->value = nv; + rc = 0; + } else if (rc == 0 || rc == -ENOENT) { + rc = 0; /* not built yet -- add_config's caller owns creation */ + } + /* else: key_find already returns a negative errno -- keep it */ + unlock(c->configmgr); + + return rc; +} + +/* cluster_build_config is idempotent: boot (xce_config_init's + * HOOK_DETAIL_CLUSTER_INIT path) creates the confvar entries via add_config; + * a re-run after a config activation (config_repl.c's activated-seam call) + * REFRESHES them instead of failing on EEXIST. The confvars are what + * collect_confvars() exports to the scheduler/worker on every config push, + * so without this refresh a live config change to heartbeat_interval/ + * inactive_heartbeat_count would never reach a running scheduler. */ int cluster_build_config(cluster_t* c) { char valbuf[128]; @@ -1470,9 +1523,11 @@ int cluster_build_config(cluster_t* c) snprintf(valbuf, sizeof(valbuf) - 1, "%d", c->myconf->cluster_config->heartbeat_interval); add_config(c, "heartbeat_interval", valbuf); + (void)config_refresh(c, "heartbeat_interval", valbuf); snprintf(valbuf, sizeof(valbuf) - 1, "%d", c->myconf->cluster_config->inactive_heartbeat_count); add_config(c, "inactive_heartbeat_count", valbuf); + (void)config_refresh(c, "inactive_heartbeat_count", valbuf); return 0; } diff --git a/src/controller/config_repl.c b/src/controller/config_repl.c index dbe10f25..de5b0685 100644 --- a/src/controller/config_repl.c +++ b/src/controller/config_repl.c @@ -335,8 +335,19 @@ int config_activate_committed(cluster_t *c) if (lrc != 0 && lrc != -ENOENT) WARN("config_activate_committed: no committed marker; " "load_primary_config fallback rc=%d", lrc); - if (lrc == 0) + if (lrc == 0) { reconcile_config_queues(c, 1); /* bug-1469 */ + /* admin-plane-gaps 1.1: same confvar refresh + live-scheduler + * re-push as the hash-verified activation path below (this + * fallback genuinely activates locally-held bytes on recovery, + * so the same materialization applies). */ + (void)cluster_build_config(c); + mylock_lock(&(c->sched.lock)); + if (c->sched.ep != NULL) { + (void)xcesched_push_configs(c); + } + mylock_unlock(&(c->sched.lock)); + } return 0; } if (rc != 0) @@ -387,6 +398,56 @@ int config_activate_committed(cluster_t *c) /* bug-1469: materialize queues the new config added (they used to need * a restart -- build_queues() only ran at boot). */ reconcile_config_queues(c, 1); + + /* admin-plane-gaps 1.1 (config reachability): materialize the rest of + * the activated configuration into live runtime state and re-push it to + * the connected scheduler, so a running scheduler picks the change up + * WITHOUT re-joining: + * 1. cluster_build_config re-runs its boot-time confvar seeding, now + * refresh-in-place (config_refresh, cluster.c) -- the confvars the + * scheduler/worker config pushes export (heartbeat_interval/ + * inactive_heartbeat_count) were previously frozen at their + * boot-time values. + * 2. reconcile_config_queues above already refreshes EXISTING queues + * in place (queue.c's queue_refresh_def) and notifies the + * scheduler/apiserver per updated/created queue. + * 3. xcesched_push_configs sends the full conf4sched (confvars, + * queues, resources, shared resvals, ugroups/hgroups/limits/ + * hproviders) to the live scheduler. The receive side is fully + * hot-apply capable (handler.c -> sched_stash_config -> main.c's + * round-boundary sched_apply_pending_config -> sched_init_config's + * in-place queue update, bug-1886); only this controller-side + * trigger was missing -- the push used to happen on scheduler join + * ONLY (sched.c on_scheduler_join), so a running scheduler kept + * its boot-time config until it re-joined. + * + * Lock ordering: this call runs while persist_apply_pump_run holds + * apply_pump_lock (the established WAL_CONFIG_MARKER apply seam, see + * the task-5 comment above). Taking c->sched.lock here follows the + * SAME apply_pump_lock -> sched.lock order this seam already + * exercises (the WAL_QUEUE_CTRL case's hook fire reaches sched_queue_ + * update, which takes c->sched.lock under the same held apply_pump_ + * lock), so no new lock-order edge and no reverse edge exists + * (nothing holding c->sched.lock ever acquires apply_pump_lock -- + * xce_pushconfig_req only enqueues on the endpoint outq). The push + * itself is a non-blocking enqueue, the same I/O class this seam + * already performs. */ + (void)cluster_build_config(c); + + mylock_lock(&(c->sched.lock)); + if (c->sched.ep != NULL) { + int prc = xcesched_push_configs(c); + if (prc < 0) { + WARN("config activation tag=%016" PRIx64 ": scheduler config " + "push failed rc=%d -- scheduler keeps its previous config " + "until its next join or config push", marker_tag, prc); + } else { + INFO("config activation tag=%016" PRIx64 ": re-pushed configs " + "to the connected scheduler", marker_tag); + } + } + mylock_unlock(&(c->sched.lock)); + return 0; } diff --git a/src/controller/include/cluster.h b/src/controller/include/cluster.h index 6a3219fd..dcd83cbc 100644 --- a/src/controller/include/cluster.h +++ b/src/controller/include/cluster.h @@ -33,6 +33,11 @@ extern int cluster_start(cluster_t*); extern int cluster_run(cluster_t*); extern void cluster_fini(cluster_t*); extern int register_types(cluster_t*); +/* admin-plane-gaps 1.1: create (boot) or refresh (config activation) the + * confvar entries the scheduler/worker config pushes export. Idempotent -- + * safe to call from both the boot hook and config_activate_committed's + * activated seam. */ +extern int cluster_build_config(cluster_t*); extern int worker_ctrl(cluster_t*, xce_hostctrl_t*); extern xce_resource_t* get_resource(cluster_t*, char*); extern int add_to_xlist(xlist_t*, void*); @@ -101,17 +106,28 @@ extern int do_job_started(void*, uint64_t, int, int*, int, int*); extern queue_t* create_queue(cluster_t*, xce_queue_t*); extern int build_queues(void*); /* bug-1469: create every queue in the active config that has no runtime - * object yet (existing ones are left untouched); with `notify`, tell the - * scheduler/apiserver about each new queue without appending a WAL fact - * (config queues are config-owned, built on every member). Returns the - * number created. */ + * object yet; with `notify`, tell the scheduler/apiserver about each + * created-or-refreshed queue without appending a WAL fact (config queues + * are config-owned, built on every member). admin-plane-gaps 1.1: an + * EXISTING runtime queue is no longer left untouched -- its config-owned + * fields are refreshed in place (queue_refresh_def) so a config activation + * that edits a queue hot-applies on the controller too. Returns the + * number created (refreshed queues are reported via TRACE, not the return). */ extern int reconcile_config_queues(cluster_t*, int notify); +/* admin-plane-gaps 1.1: (re)build/refresh the confvar entries the + * scheduler/worker config pushes export. Idempotent -- creates on first + * run (boot), refreshes in place on later runs (config activation). */ +extern int cluster_build_config(cluster_t*); extern void queue_fini(queue_t*); extern int queue_ctrl(cluster_t*, xce_queuectrl_t*); extern int xce_update_data(endpoint_t*, cluster_t*, int, int, int*, int, xce_kv_t*); extern int on_scheduler_join(cluster_t*); +/* admin-plane-gaps 1.1: (re)send the full conf4sched config push to the + * connected scheduler. Caller must hold c->sched.lock. No-op (0) when no + * cluster config is loaded yet. */ +extern int xcesched_push_configs(cluster_t*); extern int on_apiserver_join(cluster_t*, endpoint_t*); extern int xce_push_reslist(endpoint_t*, cluster_t*, int, xce_resource_t**); diff --git a/src/controller/queue.c b/src/controller/queue.c index 0f74f3cf..7efed268 100644 --- a/src/controller/queue.c +++ b/src/controller/queue.c @@ -47,10 +47,117 @@ int build_queues(void* c) return 0; } -/* bug-1469: see cluster.h. */ +/* queue_refresh_def: apply a queue-definition's CONFIG-owned fields to an + * existing runtime queue, in place (the controller-side twin of the + * scheduler's bug-1886 in-place re-apply in sched_init_queues()/ + * core_upsert_queue()). Called under `q` RTTI_LOCK_WRITE. + * + * Ownership notes (mirrors qobj.c's queue_init, which built these fields + * originally): + * - resreq/pre_exec/post_exec are interned xstrings -- dereference the + * old, makeString the new (never free()). + * - users/hosts/thresholds/windows are decoded capnp payloads owned by + * the queue -- free with the generated free_* helpers, replace with + * the NEW definition's payloads via MOVE_PTR (the config's + * cluster_config owns the old copies and is retired by + * load_primary_config after activation; the runtime queue must not + * alias into it). + * - status/priority/accept_interval/sched_delay/attrs are scalars -- + * plain copies (config is the source of truth; a re-activation that + * changed an admin state bit in the config applies it, and later WAL + * re-applies of queue_ctrl facts still win on replay order). + * + * Returns 0, or -ENOMEM when an interned-string allocation failed (the + * queue keeps its previous definition's strings in that case -- scalars + * may already have been copied, which is benign: the next activation + * re-applies). */ +static int queue_refresh_def(queue_t* q, xce_queue_t* qdef) +{ + char* ns; + + ns = makeString(qdef->resreq); + if (ns == NULL && qdef->resreq != NULL) { + return -ENOMEM; + } + if (q->queue.resreq != NULL) { + dereference(q->queue.resreq); + } + q->queue.resreq = ns; + + q->queue.attrs = qdef->attrs; + q->queue.status = qdef->status; + q->queue.priority = qdef->priority; + q->queue.accept_interval = qdef->accept_interval; + q->queue.sched_delay = qdef->sched_delay; + + if (q->queue.users != NULL) { + free_GroupConfig_ptr(&(q->queue.users)); + } + MOVE_PTR(q->queue.users, qdef->users); + + if (q->queue.hosts != NULL) { + free_GroupConfig_ptr(&(q->queue.hosts)); + } + MOVE_PTR(q->queue.hosts, qdef->hosts); + + if (q->queue.nthresholds > 0 && q->queue.thresholds != NULL) { + free_Threshold_list(q->queue.nthresholds, q->queue.thresholds); + q->queue.thresholds = NULL; + } + q->queue.nthresholds = qdef->nthresholds; + MOVE_PTR(q->queue.thresholds, qdef->thresholds); + + if (q->queue.runwindow_config != NULL) { + free_TimeWindow_ptr(&(q->queue.runwindow_config)); + } + MOVE_PTR(q->queue.runwindow_config, qdef->runwindow_config); + + if (q->queue.dispatchwindow_config != NULL) { + free_TimeWindow_ptr(&(q->queue.dispatchwindow_config)); + } + MOVE_PTR(q->queue.dispatchwindow_config, qdef->dispatchwindow_config); + + ns = makeString(qdef->pre_exec); + if (ns == NULL && qdef->pre_exec != NULL) { + return -ENOMEM; + } + if (q->queue.pre_exec != NULL) { + dereference(q->queue.pre_exec); + } + q->queue.pre_exec = ns; + + ns = makeString(qdef->post_exec); + if (ns == NULL && qdef->post_exec != NULL) { + return -ENOMEM; + } + if (q->queue.post_exec != NULL) { + dereference(q->queue.post_exec); + } + q->queue.post_exec = ns; + + ns = makeString(qdef->autopscaler); + if (ns == NULL && qdef->autopscaler != NULL) { + return -ENOMEM; + } + if (q->queue.autopscaler != NULL) { + dereference(q->queue.autopscaler); + } + q->queue.autopscaler = ns; + + return 0; +} + +/* bug-1469: see cluster.h. Admin-plane-gaps 1.1 follow-up: an existing + * runtime queue is no longer "left untouched" on a config activation -- + * its CONFIG-owned fields are refreshed in place (queue_refresh_def) and, + * with `notify`, the live scheduler/apiserver hear about the change via + * the same NO-WAL queue-update hook a fresh queue gets (config queues are + * config-owned, so no WAL_QUEUE_CTRL fact is appended for the refresh). + * Creation semantics are unchanged. */ int reconcile_config_queues(cluster_t* c, int notify) { int created = 0; + int updated = 0; int i; if ((c == NULL) || (c->qmgr == NULL) || (c->myconf == NULL) @@ -70,7 +177,26 @@ int reconcile_config_queues(cluster_t* c, int notify) key_find(c->qmgr, (void*)qdef->queue, (void**)&q); unlock(c->qmgr); if (q != NULL) { - continue; /* existing runtime queue: left untouched */ + int rc; + + /* existing runtime queue: refresh its config-owned fields */ + lock(q, RTTI_LOCK_WRITE); + rc = queue_refresh_def(q, qdef); + unlock(q); + if (rc < 0) { + WARN("%s: failed to refresh queue %s from configuration " + "(rc=%d) -- keeping previous definition", + __func__, qdef->queue, rc); + continue; + } + updated++; + + if (notify) { + INFO("queue %s updated by configuration activation", + qdef->queue); + persist_notify_queue_updated_no_wal(c, q); + } + continue; } q = create_queue(c, qdef); @@ -87,6 +213,11 @@ int reconcile_config_queues(cluster_t* c, int notify) } } + if (updated > 0) { + TRACE("%s: %d queue(s) refreshed from configuration", __func__, + updated); + } + return created; } diff --git a/src/controller/sched.c b/src/controller/sched.c index bb22db35..33f6e1eb 100644 --- a/src/controller/sched.c +++ b/src/controller/sched.c @@ -136,7 +136,17 @@ static int xcesched_push_workerlist(cluster_t* c, xlist_t* l) return rc; } -static int xcesched_push_configs(cluster_t* c) +/* xcesched_push_configs: build+send the scheduler's conf4sched config push + * (confvars/queues/resources/shared resvals from the runtime managers + + * ugroups/hgroups/limits/hproviders from the active cluster config). + * admin-plane-gaps 1.1: no longer static -- config_repl.c's activation seam + * re-pushes configs to a CONNECTED scheduler after a config activation + * (the receive side, sched_stash_config -> sched_apply_pending_config -> + * sched_init_config, is fully hot-apply capable; only the controller-side + * trigger was missing, so a running scheduler kept its boot-time config + * until it re-joined). Caller must hold c->sched.lock, exactly like + * on_scheduler_join's push branch. */ +int xcesched_push_configs(cluster_t* c) { xce_configdata_t configdata; int rc = 0; diff --git a/src/controller/testsuite/test_config_queue_reconcile.c b/src/controller/testsuite/test_config_queue_reconcile.c index 40abe061..0f836333 100644 --- a/src/controller/testsuite/test_config_queue_reconcile.c +++ b/src/controller/testsuite/test_config_queue_reconcile.c @@ -2,8 +2,11 @@ * upgrade used to exist only after a restart, because build_queues() ran * only at boot. reconcile_config_queues() creates the missing runtime queue * objects when a config is activated, notifies the scheduler/apiserver - * (without a WAL fact -- config queues are config-owned), leaves existing - * queues untouched, and is idempotent (boot's build_queues() uses it too). */ + * (without a WAL fact -- config queues are config-owned). admin-plane-gaps + * 1.1: an EXISTING queue is no longer left untouched -- its config-owned + * fields are refreshed in place (queue_refresh_def) and it gets the same + * no-WAL notify a created queue does; and reconcile stays idempotent (boot's + * build_queues() uses it too). */ #include #include @@ -81,17 +84,21 @@ static void test_reconcile_creates_queue_added_by_config(void) TEST_ASSERT_EQUAL_INT(0, xce_add_hook(&c.hooksys, XCE_CONTROLLER_HOOK_QUEUE, cqr_queue_spy, NULL)); + /* 1 created + 1 refreshed, each with its own notify fire. */ TEST_ASSERT_EQUAL_INT_MESSAGE(1, reconcile_config_queues(&c, 1), "bug-1469: the queue added by the new config must be created"); TEST_ASSERT_NOT_NULL(cqr_find(&c, "cqr-added")); TEST_ASSERT_EQUAL_PTR_MESSAGE(before, cqr_find(&c, "cqr-existing"), - "an existing queue object must be left untouched"); - TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_cqr_queue_updates, - "the scheduler/apiserver must be told about the new queue"); + "an existing queue must be refreshed in place, not replaced"); + TEST_ASSERT_EQUAL_INT_MESSAGE(2, g_cqr_queue_updates, + "the scheduler/apiserver must hear about the created AND the " + "refreshed queue (admin-plane-gaps 1.1)"); - /* Idempotent: a second activation (or boot's build_queues) adds nothing. */ + /* Idempotent: a second activation (or boot's build_queues) adds nothing; + * the refresh re-applies the same definition to BOTH queues (2 more + * notifies), so still no new object. */ TEST_ASSERT_EQUAL_INT(0, reconcile_config_queues(&c, 1)); - TEST_ASSERT_EQUAL_INT(1, g_cqr_queue_updates); + TEST_ASSERT_EQUAL_INT(4, g_cqr_queue_updates); TEST_ASSERT_EQUAL_INT(0, build_queues(&c)); TEST_ASSERT_EQUAL_PTR(before, cqr_find(&c, "cqr-existing")); @@ -99,8 +106,83 @@ static void test_reconcile_creates_queue_added_by_config(void) dereference(c.qmgr); } +/* admin-plane-gaps 1.1: editing an EXISTING queue in the activated config + * must hot-apply onto the live runtime queue in place: scalar fields + * (status/priority/...) copied, the interned resreq string swapped, and the + * same queue_t object kept (never destroyed + recreated -- dispatchers hold + * that pointer). */ +static void test_reconcile_refreshes_existing_queue_in_place(void) +{ + kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING, + .nslots = 16, .delta = 8 }; + cluster_t c; + myconf_t conf; + cluster_config_t cfg; + xce_queue_t qa; + xce_queue_t* queues[1]; + queue_t* q; + + if (makeString((char*)"cqr-probe") == NULL) + xstring_table_init(64); + + memset(&c, 0, sizeof(c)); + memset(&conf, 0, sizeof(conf)); + memset(&cfg, 0, sizeof(cfg)); + TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX)); + c.qmgr = makeInstance(&xkvmgr_type, &str_params); + TEST_ASSERT_NOT_NULL(c.qmgr); + + memset(&qa, 0, sizeof(qa)); + qa.queue = (char*)"cqr-edit"; + qa.status = 1; + qa.priority = 10; + qa.resreq = (char*)"oldres=1"; + qa.pre_exec = (char*)"old_pre.sh"; + queues[0] = &qa; + cfg.nqueues = 1; + cfg.queues = queues; + conf.cluster_config = &cfg; + c.myconf = &conf; + + TEST_ASSERT_NOT_NULL(create_queue(&c, &qa)); + q = cqr_find(&c, "cqr-edit"); + TEST_ASSERT_NOT_NULL(q); + TEST_ASSERT_EQUAL_INT(10, q->queue.priority); + TEST_ASSERT_EQUAL_STRING("oldres=1", q->queue.resreq); + + /* The admin edits the queue in the new config version. The config object + * itself is the "activated" one here, so edit it in place (a real + * activation swaps in a fresh cluster_config that reconcile then reads; + * queue_refresh_def reads the definition but only takes ownership of the + * capnp payloads, which this scalar/string-focused test leaves NULL). */ + qa.status = 0; + qa.priority = 42; + qa.resreq = (char*)"newres=2"; + qa.pre_exec = (char*)"new_pre.sh"; + + g_cqr_queue_updates = 0; + TEST_ASSERT_EQUAL_INT(0, xce_add_hook(&c.hooksys, XCE_CONTROLLER_HOOK_QUEUE, + cqr_queue_spy, NULL)); + + TEST_ASSERT_EQUAL_INT_MESSAGE(0, reconcile_config_queues(&c, 1), + "an edited queue must refresh, not create (return counts creations)"); + TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_cqr_queue_updates, + "the refreshed queue must be announced to the scheduler/apiserver"); + + /* Same object, new values -- the in-place hot-apply contract. */ + TEST_ASSERT_EQUAL_PTR(q, cqr_find(&c, "cqr-edit")); + TEST_ASSERT_EQUAL_INT(0, q->queue.status); + TEST_ASSERT_EQUAL_INT(42, q->queue.priority); + TEST_ASSERT_EQUAL_STRING("newres=2", q->queue.resreq); + TEST_ASSERT_EQUAL_STRING("new_pre.sh", q->queue.pre_exec); + + xce_hook_fini(&c.hooksys); + dereference(c.qmgr); +} + void test_config_queue_reconcile(void) { UnitySetTestFile(__FILE__); RUN_TEST(test_reconcile_creates_queue_added_by_config); + RUN_TEST(test_reconcile_refreshes_existing_queue_in_place); } diff --git a/src/controller/testsuite/test_config_repl.c b/src/controller/testsuite/test_config_repl.c index 80ccbbd3..c1101ac3 100644 --- a/src/controller/testsuite/test_config_repl.c +++ b/src/controller/testsuite/test_config_repl.c @@ -1096,6 +1096,10 @@ static void t43_commit_marker(cluster_t *c, uint64_t tag, const uint8_t sha[32]) t42_wait_applied(&c->persist, mlsn, 2000); } +/* Capturing fake for xcesched_push_configs (defined at file bottom with the + * other cluster.c-link fakes; sched.c is deliberately not linked here). */ +extern int g_crp_push_configs_called; + /* (4.3-a) MATCH activates: a committed marker whose locally-held bytes' sha256 * MATCHES the marker activates the configuration and leaves the member UN-fenced * (R7.8). */ @@ -1139,6 +1143,122 @@ static void test_config_activate_match_activates_and_unfences(void) printf("test_config_activate_match_activates_and_unfences: PASS\n"); } +/* (4.3-a2) admin-plane-gaps 1.1: a hash-verified activation re-pushes the + * full conf4sched to a CONNECTED scheduler (xcesched_push_configs), so a + * running scheduler picks the new config up WITHOUT re-joining; and the + * push must NOT fire when activation is deferred (-EAGAIN, bytes + * mismatch/absent) -- the fence's own retry path is what drives activation + * later. Observed via the capturing xcesched_push_configs fake (above). + * c->sched.ep is wired to a zeroed fd=-1 stack endpoint (same fixture shape + * test_job_array_sched_push.c proves safe: packets queue on ep->outq + * without touching a socket; the fake never even looks at it). + * + * Note the apply pump ALREADY calls config_activate_committed during + * t43_commit_marker (apply_committed_entry's WAL_CONFIG_MARKER case), while + * sched.ep is still NULL -- so the marker-commit-driven fire does not count; + * only this test's own explicit, endpoint-wired call must. */ +static void test_config_activation_repushes_configs_to_connected_scheduler(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_crp11_push_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + cluster_t *c = t42_make_cluster(tmp); + TEST_ASSERT_NOT_NULL(c); + t42_make_leader(c, /*term*/3, /*n_peers*/1); + TEST_ASSERT_EQUAL_INT(0, + commit_engine_init(&c->commit, &c->persist, 1, 8)); + repl_set_commit_engine(&c->persist.repl, &c->commit); + myconf_t *mc = t43_attach_myconf(c, tmp); + + size_t blen = 0; + uint8_t *bytes = t43_make_config_blob(&blen); + uint8_t sha[32]; + SHA256(bytes, blen, sha); + uint64_t tag; + { uint64_t t; memcpy(&t, sha, 8); tag = be64toh(t); } + TEST_ASSERT_EQUAL_INT(0, conf_save_config(mc, bytes, blen, sha)); + free(bytes); + + /* The marker commit itself drives activation via the apply pump, while + * sched.ep is still NULL -- no push fires (and no crash from the NULL). */ + g_crp_push_configs_called = 0; + t43_commit_marker(c, tag, sha); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_crp_push_configs_called, + "no scheduler push while the scheduler is not connected"); + + /* Wire a (zeroed, fd=-1) connected scheduler endpoint and activate. */ + endpoint_t ep; + memset(&ep, 0, sizeof(ep)); + ep.fd = -1; + c->sched.ep = &ep; + + TEST_ASSERT_EQUAL_INT(0, config_activate_committed(c)); + TEST_ASSERT_EQUAL_INT_MESSAGE(1, g_crp_push_configs_called, + "admin-plane-gaps 1.1: a hash-verified activation must re-push " + "configs to the connected scheduler"); + TEST_ASSERT_EQUAL_INT(0, config_serve_fenced(c)); + TEST_ASSERT_NOT_NULL(mc->cluster_config); + TEST_ASSERT_EQUAL_HEX64(tag, mc->config_version_tag); + + /* A second activation (same committed marker, idempotent re-run) pushes + * again -- the push is unconditional on success, matching the + * on_scheduler_join push branch it mirrors. */ + TEST_ASSERT_EQUAL_INT(0, config_activate_committed(c)); + TEST_ASSERT_EQUAL_INT(2, g_crp_push_configs_called); + + c->sched.ep = NULL; + + t43_free_myconf(c); + commit_engine_fini(&c->commit); + t42_teardown(c); + printf("test_config_activation_repushes_configs_to_connected_scheduler: PASS\n"); +} + +/* (4.3-a3) admin-plane-gaps 1.1: the -EAGAIN defer path (committed marker's + * bytes absent/mismatched locally) must NOT push anything -- the member is + * fenced and waiting for bytes, so there is nothing activated to push. */ +static void test_config_activation_deferred_does_not_push_configs(void) +{ + char tmp[256]; + strcpy(tmp, "/tmp/xce_crp11_defer_XXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(tmp)); + + cluster_t *c = t42_make_cluster(tmp); + TEST_ASSERT_NOT_NULL(c); + t42_make_leader(c, /*term*/4, /*n_peers*/1); + TEST_ASSERT_EQUAL_INT(0, + commit_engine_init(&c->commit, &c->persist, 1, 8)); + repl_set_commit_engine(&c->persist.repl, &c->commit); + myconf_t *mc = t43_attach_myconf(c, tmp); + + /* Commit a marker whose bytes were NEVER saved locally (the out-of-band + * delivery has not arrived) -- the classic fence-and-retry state. */ + uint8_t shaB[32]; + uint64_t tagB = t42_make_ref(shaB, 0x77); + g_crp_push_configs_called = 0; + t43_commit_marker(c, tagB, shaB); + + endpoint_t ep; + memset(&ep, 0, sizeof(ep)); + ep.fd = -1; + c->sched.ep = &ep; + + TEST_ASSERT_EQUAL_INT_MESSAGE(-EAGAIN, config_activate_committed(c), + "absent bytes must defer activation (-EAGAIN), not push"); + TEST_ASSERT_EQUAL_INT(1, config_serve_fenced(c)); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, g_crp_push_configs_called, + "a deferred (mismatched/absent-bytes) activation must NOT push " + "configs to the scheduler -- nothing was activated"); + + c->sched.ep = NULL; + + t43_free_myconf(c); + commit_engine_fini(&c->commit); + t42_teardown(c); + printf("test_config_activation_deferred_does_not_push_configs: PASS\n"); +} + /* (4.3-b) MISMATCH fences + retries, NEVER activates: an injected hash mismatch * (locally-held bytes whose sha256 != the committed marker) keeps the member * FENCED (non-serving), ALIVE, and retrying, and does NOT activate the mismatched @@ -3629,6 +3749,19 @@ int do_runtasks(cluster_t *c, endpoint_t *ep, xce_request_t *req) int handle_apiserver_leave(cluster_t *c, endpoint_t *ep) { (void)c; (void)ep; return 0; } +/* admin-plane-gaps 1.1: config_repl.c's activation seam now calls + * xcesched_push_configs() (sched.c) -- which this binary deliberately does + * NOT link (same real-vs-fake collision the on_scheduler_join fake above + * documents). This capturing fake stands in for it: the new activation-push + * test below asserts on the call count, and every OTHER test in this file + * that drives config_activate_committed relies on the fake being a no-op + * (all t42/t43 fixtures leave c->sched.ep NULL, so the production guard + * skips the call entirely -- the counter exists purely for the one test + * that wires an endpoint in). */ +int g_crp_push_configs_called; +int xcesched_push_configs(cluster_t *c) +{ (void)c; g_crp_push_configs_called++; return 0; } + static io_packet_t *cpr22_build_config_push_reply_pkt(int accepted, uint64_t version_tag, uint64_t term) @@ -7457,6 +7590,10 @@ void test_config_repl(void) RUN_TEST(test_committed_marker_survives_checkpoint_trim); RUN_TEST(test_config_activate_match_activates_and_unfences); + /* admin-plane-gaps 1.1: activation re-pushes configs to a connected + * scheduler; the -EAGAIN defer path pushes nothing. */ + RUN_TEST(test_config_activation_repushes_configs_to_connected_scheduler); + RUN_TEST(test_config_activation_deferred_does_not_push_configs); RUN_TEST(test_config_activate_mismatch_fences_and_does_not_activate); RUN_TEST(test_config_activate_obtained_bytes_verified_before_activate); RUN_TEST(test_config_activate_recovery_uses_latest_committed_marker); diff --git a/src/frontend/apiserver/backend/clib.go b/src/frontend/apiserver/backend/clib.go index be843e68..70a38169 100644 --- a/src/frontend/apiserver/backend/clib.go +++ b/src/frontend/apiserver/backend/clib.go @@ -567,6 +567,20 @@ func handleNotifyData(nd *C.xce_notify_data_t) C.int { } gHandler.OnJobPidInfo(uint64(pi.jobid), cPidInfoToRaw(pi)) + case C.NotifyData_data_schedUpdateQueue: + // Queue open/close/priority changes reach this API server as a + // per-queue notify (queue_ctrl's hook fire, and configuration + // activation's reconcile_config_queues refresh). Without this case + // the push was silently dropped here and store.Queues kept the + // join-time snapshot (dispatchConfigPush at OnBackendConnect) until + // the next full config push -- so `xce queues list` never reflected + // a hot-applied config edit or a queue state change. + qp := (*C.xce_queue_t)(C.xce_notify_data_ptr(nd)) + if qp == nil { + return 0 + } + gHandler.OnQueueUpdated(cQueueToModel(qp)) + case C.NotifyData_data_schedUpdateResvals: rvl := (*C.xce_resval_list_t)(C.xce_notify_data_ptr(nd)) if rvl == nil { diff --git a/src/frontend/apiserver/backend/handler_test.go b/src/frontend/apiserver/backend/handler_test.go index ba6651e0..2b239c90 100644 --- a/src/frontend/apiserver/backend/handler_test.go +++ b/src/frontend/apiserver/backend/handler_test.go @@ -448,6 +448,27 @@ func TestOnClusterUpdated_PartialMerge(t *testing.T) { } } +func TestOnQueueUpdated_InPlaceRefresh(t *testing.T) { + resetStores() + h := DefaultHandler{} + + // Join-time config push seeded the queue table. + h.OnQueueUpdated(&models.QueueInfo{QueueName: "short", Priority: 45}) + + // A later per-queue notify (queue open/close or a hot-applied config + // activation) refreshes the SAME key in place — `xce queues list` must + // reflect the new priority without an apiserver restart. + h.OnQueueUpdated(&models.QueueInfo{QueueName: "short", Priority: 46}) + + got, ok := store.Queues.Get("short") + if !ok { + t.Fatal("queue short should exist in store after OnQueueUpdated") + } + if got.Priority != 46 { + t.Errorf("Priority: got %d, want 46 (notify must refresh in place)", got.Priority) + } +} + func TestOnHostUpdated_PartialMerge(t *testing.T) { resetStores() h := DefaultHandler{}