diff --git a/.kiro/specs/external-license-resources/brief.md b/.kiro/specs/external-license-resources/brief.md index d15d266c..0daf8973 100644 --- a/.kiro/specs/external-license-resources/brief.md +++ b/.kiro/specs/external-license-resources/brief.md @@ -12,7 +12,7 @@ XCE 无法调度依赖外部 license server（FlexLM/lmstat 等）的 EDA 应用 XCE_RESF_EXTERNAL 是死 flag）。 ## Desired Outcome -- **xceextresd**：独立采集 daemon（避免 lmstat 阻塞连坐 controller）， +- **xceesd**：独立采集 daemon（避免 lmstat 阻塞连坐 controller）， 以新 `HostRole` 成员 join controller，采集器插件：flexlm（lmstat）、 script（任意外部资源）。wire 零 schema 变更（复用 ResValList 管线）， 唯一变更是 HostRole 加成员。 @@ -38,7 +38,7 @@ share 机制）。 - **elastic-shared-resources**：同管线两种记账模式按资源分流 （EXTERNAL=server-authoritative，无 flag=elastic base+borrowed）—— opupd 守卫"SHARED 即收"预留其写入路径。即对其决策日志 P9 的回答。 -- **PR-2 proxy**（resource-questions.md）：xceextresd 定位为该 proxy 的 +- **PR-2 proxy**（resource-questions.md）：xceesd 定位为该 proxy 的 第一片（只读 report 能力），HostRole 只加一个成员。 - **project-config**：per-app 配额→per-project share（project tree 是 唯一并发上限处，limits.yaml 退役）。需要其扩展：集群级 pool location。 @@ -48,7 +48,7 @@ share 机制）。 ## Boundary Candidates - HostRole_extres join/leave + opupd 准入守卫（controller） - unsatisfy_ EXTERNAL 分支（scheduler） -- xceextresd daemon 与采集器插件（新 src/extresd/） +- xceesd daemon 与采集器插件（新 src/esd/） - mapping.py `location: [external]` 路由 ## Out of Boundary @@ -64,6 +64,7 @@ share 机制）。 provisioning-demand（headroom 需求源）、project-config（集群级池） ## Open（待用户/跨团队决策） -1. 命名：xceextresd（推荐）vs xcelsd。 +1. ~~命名~~ 已定（2026-10-27）：**xceesd**（src/esd/，esd.yaml；HostRole + 枚举成员 extres）。 2. 断联策略：fail-open（推荐）vs fail-closed。 3. P9 模式拆分与三个扩展点需与 Rongsong 确认（design §3.6）。 diff --git a/.kiro/specs/external-license-resources/design.md b/.kiro/specs/external-license-resources/design.md index 7f16e2c9..5dff37ce 100644 --- a/.kiro/specs/external-license-resources/design.md +++ b/.kiro/specs/external-license-resources/design.md @@ -110,17 +110,23 @@ license 服务器的"当前空闲数"是外部真值；本地只需补一个窗 ### 3.1 架构决策：独立 daemon + 采集器插件 -**xceextresd（外部资源守护进程）**，新 `src/extresd/`，C 实现， -照 scheduler 的 join 方式以新 `HostRole_extres` 角色接入 controller。 -内部采集器插件接口： +**xceesd（外部资源守护进程；XCE External-resources Daemon）**，新 +`src/esd/`，C 实现，照 scheduler 的 join 方式以新 `HostRole_extres` +角色接入 controller。内部采集器插件接口： ``` -xceextresd +xceesd ├─ collector/flexlm.c — lmstat -c port@host -f feature 解析（EDA license 主路径） ├─ collector/script.c — 用户脚本输出 key=value/JSON（任意外部资源：云配额、网络带宽…） └─ collector/http.c —（预留） ``` +> **命名**（2026-10-27 定）：daemon 名 `xceesd`，目录 `src/esd/`，二进制 +> `xceesd`，配置 `esd.yaml`，代码标识符 `esd`。wire 层 HostRole 枚举成员 +> 仍用 `extres`（角色语义，与 reporter/provisioner 讨论对齐）。 +> 注意 `esd` 是高频子串（closed/parsed…），全文检索认准全名。 +> 旧稿中的 `xceextresd`/`extresd` 均指本 daemon（改名前写法）。 + - **为什么不是 controller 内插件**：lmstat 是秒级阻塞调用，违反 controller async I/O 红线（delaymgr/vtask 不得阻塞，HA/心跳会被连坐）；license server 故障必须与 controller 故障域隔离。 @@ -133,7 +139,7 @@ xceextresd ### 3.2 数据流（wire 零 schema 变更，除 HostRole 加成员） ``` -xceextresd (join: HostRole_extres) +xceesd (join: HostRole_extres) │ OPCode_update / Update_data_resvals（现 wire：ResourceVal{name, host:"", val}） ▼ opupd.c do_update_resvals ── 放行 extres 角色；flag 守卫只许 SHARED 资源（§3.6） @@ -185,7 +191,7 @@ EXTERNAL：F ≥ uncomit + wanted（unsatisfy_ 循环内一个 flag 分支； 与 Slurm `_handle_consumed` 的 deficit 算法同构（都是"避开外部消费者份额"）， 但更简：不需要 LastConsumed/external 推导，因为 lmstat 直接给 free。 -### 3.3b elim（LSF load index）兼容性 —— 已盘点，归 xcecd 不归 extresd +### 3.3b elim（LSF load index）兼容性 —— 已盘点，归 xcecd 不归 esd **LSF elim 模型**：用户在主机上部署脚本，周期 echo `资源名 数量`，lim 收集后 变成可调度资源。per-host 语义。License Scheduler 历史上正是从 elim 派生—— @@ -217,15 +223,15 @@ posix_spawn 路径；读 stdout `name value` 行 elim 原格式；调现有 regi API 进 registry；脚本超时看门狗 kill）。MAX_LOAD_INDICES=64 够用。 工作量 D 类（中型），与 E2 独立可并行。 -**归属决策：elim 支持放 xcecd，不放 xceextresd**。理由： +**归属决策：elim 支持放 xcecd，不放 xceesd**。理由： 1. 作用域——elim 值 per-host（"这台机器有多少"），license 值集群全局 - （"全池还剩多少"）；worker 本来就站在每台主机上，让 extresd 管主机值 + （"全池还剩多少"）；worker 本来就站在每台主机上，让 esd 管主机值 是数据绕远路。 2. 故障域——elim 脚本挂只影响一台主机上报；license server 断联影响全集群 准入。xcecd 采集器继承前者的隔离性。 3. LSF 自己也是分开的（lim vs lsld/blcollect），复刻这个分界。 4. XCE 架构两条腿天然对齐：worker→per-worker resvals→host 判式； - extresd→sharedinst→全局判式。同一资源模型（定义/rusage/pend reason/ + esd→sharedinst→全局判式。同一资源模型（定义/rusage/pend reason/ CLI 展示）。 **统一记账规则（两条腿共享，同一 flag 位决定）**：EXTERNAL 资源的满足性 @@ -252,13 +258,13 @@ count) 聚合——无法判定 A=2/B=3 还是 A=3/B=2。LSF License Scheduler daemon 每轮对账：Σ(派发中) vs 服务器 used，差值 = 外部消费者 + 泄漏， 异常 alert。漂移有界且方向安全：少拿→配额保守不超卖；多拿→服务器 total 暴露。 -3. **报告层 = 证据分级 + 残差桶**（matcher 为 extresd 插件，不猜、显式 +3. **报告层 = 证据分级 + 残差桶**（matcher 为 esd 插件，不猜、显式 化"不知道"）： - Tier 0：派发记账（app→授予数），精确，永远可用； - Tier 1：vendor checkout 属性（如 Synopsys LM_PROJECT 类，lmstat 可见； XCE 作业 env 已注入 XCE_JOBID，vendor 走 env 即可盖入），精确； - Tier 1'：PID 匹配（部分 vendor daemon 报客户端 PID）——jobPidInfo - 已有 jobid→pid 映射（3545aa0d），extresd 从 controller 拉即可，精确； + 已有 jobid→pid 映射（3545aa0d），esd 从 controller 拉即可，精确； - Tier 2：(user, host) + checkout start 时间窗 匹配作业记录（LSF 基线 法升级：两个独立 XCE 作业通常可按 start 时间拆开），user+host 相同时 只能给出"这批作业合计持有 K"的界； @@ -287,7 +293,7 @@ resources: ``` ```yaml -# extresd.yaml（daemon 侧：操作） +# esd.yaml（daemon 侧：操作） poll_interval: 30s resources: - name: Verilog_XL # 必须与 resources.yaml 同名（控制器侧 unknown-resource 拒） @@ -300,7 +306,7 @@ resources: stale_limit: 150s # 连续取不到 → WARN 保旧值（fail-open，可配 fail-closed 清零） ``` -职责分界：resources.yaml 管声明（作业能否 rusage），extresd.yaml 管采集 +职责分界：resources.yaml 管声明（作业能否 rusage），esd.yaml 管采集 （值从哪来）。`seed_sharedinst_from_config` 对 EXTERNAL 资源**不播种** （值只来自 daemon；daemon 未上线时 get_resvals 读 0 → PEND NO_RESOURCE detail 带资源名）。 @@ -317,7 +323,7 @@ detail 带资源名）。 3. resource.c unsatisfy_：EXTERNAL 跳过 inuse 项（+trace 行） 4. cluster.c seed：EXTERNAL 不播种 5. mapping.py：`location: [external]` 路由 -6. xceextresd 骨架：join/rejoin/failover 重连即全量重推（worker 状态补报 +6. xceesd 骨架：join/rejoin/failover 重连即全量重推（worker 状态补报 9d7628e 同款语义）+ script/flexlm 采集器 + 轮询去抖 7. TDD：testsuite 红→绿（opupd 守卫、unsatisfy_ 分支、采集器解析）； t1 E2E 用 script 采集器 + 假 license 脚本（不必有真 FlexLM） @@ -335,7 +341,7 @@ PEND detail 含 lic 名）；daemon kill/重启后值全量补推；controller daemon 重新 join 新主并补推；`--sim` 录制含 daemon 推送事件且回放一致。 **elim 归属（与 E2 并行的独立 D 类项）**：LSF elim（per-host 外部资源脚本） -归 xcecd（worker）不归 xceextresd——详见 §3.3b。license（全局）与 elim +归 xcecd（worker）不归 xceesd——详见 §3.3b。license（全局）与 elim （主机）是两条腿、同一资源模型、同一记账规则（EXTERNAL 跳 inuse）。 ### 3.6 与 Rongsong 统一资源模型 spec 群的合并性审查（2026-10-26） @@ -379,13 +385,13 @@ daemon 推 free 直接覆盖 sharedInst。两套各写各的 → 同一资源 sh 唯一并发上限处，limits.yaml 退役"（332f86ea）；§3.3c Tier-0 配额写成 project share/member_quota 由 limit plugin 编译，不发明新机制。 - staleness/interval 进**资源描述符**（其 Phase 1 descriptor 字段 - interval/last_report 已预留），不放 extresd.yaml；daemon 只执行。 + interval/last_report 已预留），不放 esd.yaml；daemon 只执行。 - 持久化分级：daemon 推的 F 是 dynamic reported value → 非 WAL fact、 进 scheduler snapshot + daemon 重连全量补推即安全；将来 elastic 模式 的 total 才是 WAL fact。与 persistence.md 一致。 - 角色收敛：PR-2 proxy（leaning）统一 reporter+provisioner 为一个 daemon role；**HostRole 只加一个成员**（避免将来再出现第二个同义 enum）， - xceextresd 定位为 PR-2 proxy 的第一片（只读 report 能力）。 + xceesd 定位为 PR-2 proxy 的第一片（只读 report 能力）。 - 归因 Tier 1' PID 匹配引用其 jobPidInfo（我们已实施，3545aa0d）。 **我们的设计对其 spec 的三个扩展点**（他目前装不下，需带回讨论）： @@ -407,8 +413,8 @@ daemon 推 free 直接覆盖 sharedInst。两套各写各的 → 同一资源 sh "or reuse the provisioner role"）。 ### 3.7 提请决策（未决） -1. daemon 命名：`xceextresd`（通用外部资源，推荐）vs `xcelsd`（贴 LSF - license 语义）；定位为 PR-2 proxy 第一片（见 §3.6）。 +1. ~~daemon 命名~~ 已定（2026-10-27）：`xceesd` / `src/esd/` / `esd.yaml`； + HostRole 枚举成员仍为 `extres`；定位为 PR-2 proxy 第一片（见 §3.6）。 2. 断联默认策略：fail-open（保旧值+stale WARN，推荐）vs fail-closed（清零停派）。 3. P9 模式拆分与三个扩展点需与 Rongsong 确认（§3.6）。 diff --git a/.wolf/buglog.json b/.wolf/buglog.json index d4f2c157..118a9a92 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -6,7 +6,7 @@ "id": "bug-1971", "status": "fixed", "status_updated": "2026-10-22", - "status_note": "", + "status_note": "two-choke archive in xce-build.sh; test harness green", "component": "build", "severity": "wrong-result", "timestamp": "2026-10-22T07:50:00Z", @@ -27,6 +27,198 @@ "68c3ccd7" ], "record": ".wolf/bugs/bug-1971.md" + }, + { + "id": "bug-1972", + "status": "fixed", + "status_updated": "2026-10-26", + "status_note": "", + "component": "controller", + "severity": "crash", + "timestamp": "2026-10-26T09:46:58Z", + "last_seen": "2026-10-26T09:46:58Z", + "occurrences": 4, + "error_message": "test_job_live_gc asserted destruction by reading freed memory\n\n**Observed:**\n`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job`\n(test_job_live_gc.c:169):\n\n```\n==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290\nREAD of size 8 at 0x5110000da290 thread T0\n #0 job_get_refcount ../lib/jobj.c:715\n #1 test_sweep_purges_expired_done_job test_job_live_gc.c:169\nfreed by:\n #3 job_dereference ../lib/jobj.c:705\n #5 job_live_gc_sweep ../controller/job.c:4777\n```\n\n**Expected:** a correct sweep dereferences the purged job down to refcount 0\nand destroys it; the test must verify destruction without touching the freed\nobject.\n\n**Impact:** controller suite aborts (killed by SIGABRT); the whole\n`controller test` meson entry fails. Test-only — the production sweep\n(job.c:4765-4777, double dereference: collection ref + table's makeJob ref)\nis correct.\n\n**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier\nruns all predate that commit).", + "file": "src/controller/testsuite/test_job_live_gc.c", + "root_cause": "The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs\nAFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the\nentire RTTI object (header + private + data in one calloc block). Reading the\nrefcount through the dangling `job_t*` is itself the use-after-free: the\nstronger the sweep does its job, the more surely the test crashes. The same\npattern is absent from test_apply_pump.c's refcount test because there the\njob is never destroyed.", + "fix": "Move the refcount probe BEFORE the sweep and invert its meaning: assert\nrefcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`,\nthen verify destruction by absence — `jlg_find_job(81001) == NULL` plus the\nidempotent second sweep (both already present). `expired` is never dereferenced\nafter the sweep; a comment marks the last safe point.\n\nVerification:\n- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic\n ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero\n ASAN reports).\n- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0);\n sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue,\n bug-free fixture staleness from fc33a5da's schema change, handled in the same\n working tree).\n- Live: n/a (test-only defect; production sweep unchanged).", + "tags": [ + "test-only", + "uaf", + "gc" + ], + "related_bugs": [], + "fix_commits": [ + "f780a166" + ], + "record": ".wolf/bugs/bug-1972.md" + }, + { + "id": "bug-1973", + "status": "fixed", + "status_updated": "2026-10-27", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-27T03:27:47Z", + "last_seen": "2026-10-27T03:27:47Z", + "occurrences": 1, + "error_message": "cluster_build_resources aborted at first EEXIST: new resource defs never registered\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).", + "file": "src/controller/cluster.c, src/controller/testsuite/test_config_resource_reconcile.c", + "root_cause": "`cluster_build_resources()` looped over the decoded config's resource list\ncalling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)\nreturns NULL with errno=EEXIST when the name is already registered — which is\nthe COMMON case at activation (boot already registered some definitions). The\nold loop treated that rc as fatal and aborted the whole loop, so every\ndefinition AFTER the first already-registered one was never registered.", + "fix": "Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under\none write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;\nRES_APPLY_CHANGED updates the registered object's definition fields\n(valtype/ordtype/flags/interval) IN PLACE — none of them touches the\nregistration slot (idx) or accumulated accounting; mirrors\nreconcile_config_queues' in-place refresh precedent (second live bug, see\nbug-1974 — kept here as one loop). Negative rc still propagates.\n\nVerification:\n- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +\n `test_reconcile_resources_hot_applies_changed_definition`\n (src/controller/testsuite/test_config_resource_reconcile.c); RED before the\n fix (lic1 unregistered / flags stayed 0x00), GREEN after.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in\n sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at\n free=1; release re-admitted it.", + "tags": [ + "config", + "reconcile", + "E2", + "external-license" + ], + "related_bugs": [ + "bug-1854" + ], + "fix_commits": [ + "2fcacac2" + ], + "record": ".wolf/bugs/bug-1973.md" + }, + { + "id": "bug-1974", + "status": "fixed", + "status_updated": "2026-10-27", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-27T03:35:12Z", + "last_seen": "2026-10-27T03:35:12Z", + "occurrences": 1, + "error_message": "config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM\n\n**Observed:**\nController log on t1 after re-pushing a corrected cluster.bin (extlic mapped\nwith `location: [external]` → flags SHARED|GLOBAL|EXTERNAL):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted\n```\n\n**Expected:** update_val_res' SHARED branch routes a host==\"\" resval for a\nSHARED-flagged resource into c->sharedinst; with correct flags the push lands.\n\n**Impact:** the extres daemon's pushes were permanently refused on a controller\nwhose registration predated the flag fix — the stale shape was frozen for the\nprocess lifetime, so re-pushing a corrected config did not heal it.\n\n**Found by:** E2 Phase 1 live E2E on t1.", + "file": "src/controller/cluster.c, src/controller/worker.c, src/xceadmin/src/xcecommon/mapping.py", + "root_cause": "Two compounding factors:\n1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/\n xceadmin-venv) carries a stale mapping.py with NO resourcemap handling\n (83-line diff vs tree) — cluster bins it builds carry flags=0 for every\n resource. Ops issue, fixed by building with the tree venv\n (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs\n redeployment.\n2. **Controller**: even with a corrected bin, `cluster_build_resources()`\n treated an already-registered name as \"nothing to do\" (bug-1854's\n RES_APPLY_CHANGED was reported but never applied), so the stale flags=0\n shape survived every activation.", + "fix": "`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the\nregistered object's definition fields (valtype/ordtype/flags/interval) in\nplace under the resmgr write lock — none of them touches the registration\nslot (idx) or accumulated accounting; mirrors reconcile_config_queues'\n\"updated by configuration activation\" in-place refresh precedent. (Same loop\nas bug-1973's fix.) Producer side tracked separately as ops follow-up:\nredeploy the xceadmin venv.\n\nVerification:\n- Test: `test_reconcile_resources_hot_applies_changed_definition`\n (src/controller/testsuite/test_config_resource_reconcile.c); RED before\n (flags stayed 0x00), GREEN after (0x1C, idx preserved).\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push\n landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN.", + "tags": [ + "config", + "flags", + "E2", + "external-license" + ], + "related_bugs": [ + "bug-1854", + "bug-1973" + ], + "fix_commits": [ + "2fcacac2" + ], + "record": ".wolf/bugs/bug-1974.md" + }, + { + "id": "bug-1975", + "status": "fixed", + "status_updated": "2026-10-27", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-27T03:40:05Z", + "last_seen": "2026-10-27T03:40:05Z", + "occurrences": 1, + "error_message": "xceesd idle on dead connection after controller restart (stable value: no re-push)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).", + "file": "src/esd/src/esd.c, src/lib/client.c, src/lib/client_internal.h", + "root_cause": "Three mechanisms conspired:\n1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip\n push) — with a stable value it pushes NOTHING.\n2. A reconnect inside the client library only fires while INSIDE a push's\n failover call — no pushes, no reconnect.\n3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;\n with no push attempt it never runs.", + "fix": "`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`\n(bug-1760's pure predicate — the transport's RDHUP handler sets\nXCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.\nThe write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the\nfull re-push contract for the failover path.\n\nVerification:\n- Test: covered by the live contract (client-side predicate already unit\n tested by bug-1760's suite); esd-side loop is integration-shaped (the\n trigger is a transport callback), so live E2E is the proof.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — controller killed and restarted; esd logged\n `connection broken -- forcing a full re-push to reconnect`, rotated to the\n new primary, re-pushed; `xcecli resources distribution list` shows extlic=4\n and a fresh rusage[extlic=3] job RUNS.", + "tags": [ + "esd", + "external-license", + "reconnect", + "E2" + ], + "related_bugs": [ + "bug-1760", + "bug-1973", + "bug-1974" + ], + "fix_commits": [ + "2fcacac2" + ], + "record": ".wolf/bugs/bug-1975.md" + }, + { + "id": "bug-1976", + "status": "fixed", + "status_updated": "2026-10-28", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-28T05:37:09Z", + "last_seen": "2026-10-28T05:26:00Z", + "occurrences": 1, + "error_message": "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced\n\n**Observed:**\nE2 Phase 3 t1 E2E: 4 projA jobs with `-R \"rusage[extlic=1]\"` against a\nper-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran\ninstead of the 4th PENDing with reason limit_global. Live gdb on the\nscheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with\n3 projA jobs RUN: the vres charge never landed.\n\n**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job\nre-admits it; projB unaffected (own per-project instance).\n\n**Impact:** every scheduler-side global limit (the limit vres mechanism)\nwas advisory only — no quota was ever enforced, for ANY shared resource\ncarried via allocs_shared, since dispatch-time adoption overwrote the\nscheduler's own charged accounts with a shared-stripped copy.\n\n**Found by:** spec task (external-license-resources E2 Phase 3 E2E).", + "file": "src/controller/job.c, src/controller/testsuite/test_dispatch_commit.c", + "root_cause": "The controller's dispatch-time commit (`dispatch_a_job_()`, job.c)\nrebuilt `j->job.allocs` from the scheduler's dispatch payload but\nmigrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs,\nallocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared`\nwas neither moved nor freed (a leak on top). That rebuilt record is\nwhat `sched_job_update_fun()` re-sends to the scheduler as ground\ntruth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch,\ncore_events.c) rebuilds `job->accounts` from exactly that wire copy —\nso `accounts_shared == NULL`, `sched_account_jobs` charges nothing to\nthe vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's\nown commit (`sched_commit_decision`, scheduler.c:1806-1812) built the\naccounts correctly — the controller's \"ground truth\" clobbered it.", + "fix": "job.c dispatch-time commit: migrate the shared records alongside the\nhost records — `MOVE_PTR(j->job.allocs->allocs_shared,\nalloc->allocs_shared); j->job.allocs->nres_shared =\nalloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the\ncaller frees the payload; also closes the leak).\n\nVerification:\n- Test: `test_dc_success_migrates_shared_allocs`\n (src/controller/testsuite/test_dispatch_commit.c). RED before the fix\n (\"Expected 2 Was 0\" on nres_shared), GREEN after.\n- Suites: full meson test 136/136 OK (build.dbg), including controller\n unit suite 1361/0.\n- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill\n one RUN job → pending projA re-admitted, external free pool\n (fake_lic) enforced independently; all daemons healthy throughout.", + "tags": [ + "external-license-resources", + "limits", + "shared-resources", + "dispatch", + "e2-phase3" + ], + "related_bugs": [ + "bug-1860" + ], + "fix_commits": [ + "76ddd75f" + ], + "record": ".wolf/bugs/bug-1976.md" + }, + { + "id": "bug-1977", + "status": "fixed", + "status_updated": "2026-10-28", + "status_note": "", + "component": "controller", + "severity": "crash", + "timestamp": "2026-10-28T11:56:58Z", + "last_seen": "2026-10-28T11:56:58Z", + "occurrences": 1, + "error_message": "crr_build_config callers' stack arrays smaller than builder -- ASAN stack overflow\n\n**Observed:**\n```\nERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc...\nWRITE of size 8 at 0x7ffc... thread T0\n #0 crr_build_config testsuite/test_config_resource_reconcile.c:126 (dists[3] = &d4)\n #1 test_reconcile_resources_registers_and_seeds_shared_pools testsuite/test_config_resource_reconcile.c:161\n```\nEvery test in the file that calls `crr_build_config` crashes the same way;\nthe full controller suite reports 137/137 FAIL (meson reports each fixture\ncrash as its own failure).\n\n**Expected:** all four callers pass arrays large enough for the builder's\nfixtures; the suite stays green.\n\n**Impact:** the whole controller testsuite was unrunnable under ASAN since the\nexternal-license-resources test fixtures landed (commit 2fcacac2) -- any\nregression in the suite was invisible behind the deterministic crash.\n\n**Found by:** ASAN -- full regression build for the license-resource feature\n(`src/tools/xce-build.sh controller`, profile=asan).", + "file": "src/controller/testsuite/test_config_resource_reconcile.c", + "root_cause": "`crr_build_config` (test_config_resource_reconcile.c:70) registers FIVE\nresources (lic1/hostres/dyn1/extlic/slots -> writes `resources[4]`) and FOUR\ndistributions (writes `dists[3]`) and FIVE vals (writes `vals[4]`). The four\ncaller test functions declared only\n`resources[4] / dists[3] / vals[4]`, so the builder's writes to index 4 / 3 /\n4 overflowed the callers' stack arrays. When the extlic E2 tests were added to\nthe builder's fixture set (2fcacac2), none of the caller declarations were\nresized. (Earlier builds of this branch compiled only a worker-focused subset\n-- `test_config_resource_reconcile` was not linked -- so the overflow went\nunnoticed until the full controller suite was built.)", + "fix": "Resize all four caller arrays to match the builder: `resources[5]`,\n`dists[4]`, `vals[5]`, each with a bug-1977 comment pointing at the sizing\ncontract. Files: src/controller/testsuite/test_config_resource_reconcile.c\n(test_reconcile_resources_registers_and_seeds_shared_pools,\ntest_reconcile_resources_registers_new_defs_past_eexist,\ntest_reconcile_resources_hot_applies_changed_definition,\ntest_reconcile_resources_is_idempotent_and_hot_applies).\n\nVerification:\n- Test: whole-file suite `controller - xce:controller test` (meson): RED\n 137/137 before the fix, GREEN after.\n- Suites: `src/tools/xce-build.sh controller` profile=asan: 6/6 OK\n (sched push, sched worker change, apiserver push, host runusage, no-fork\n rule, controller 91s).\n- Live: n/a (testsuite-only defect; no production code touched).", + "tags": [ + "testsuite", + "asan" + ], + "related_bugs": [], + "fix_commits": [ + "8c1995c2" + ], + "record": ".wolf/bugs/bug-1977.md" + }, + { + "id": "bug-1978", + "status": "fixed", + "status_updated": "2026-10-28", + "status_note": "", + "component": "controller", + "severity": "wrong-result", + "timestamp": "2026-10-28T12:47:59Z", + "last_seen": "2026-10-28T12:47:59Z", + "occurrences": 3, + "error_message": "esd reconnect re-pushes without re-join -- pushes -EPERM forever\n\n**Observed:**\n```\nt2 /tmp/xceesd-d.log:\n[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1)\n[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s)\n[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1)\n... (every 10s poll, forever)\n```\nAfter any controller restart the daemon's pushes fail rc=-1 (-EPERM) on\nevery attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED,\nthe poll loop runs, the collector script fires every 10s, the status file\nrefreshes with `stale: false` -- only pushes_fail climbs.\n\n**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and\npushes resume (pushes_ok advances, `xce resources distribution list` shows\nextlic).\n\n**Impact:** the entire external-license value chain dies at the first\ncontroller restart: extlic never reaches c->sharedinst, the scheduler never\nhears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever\n(admission sees free=0). Found during the license-resource regression: the\nregression config push (gen 19) succeeded but `resources distribution list`\nshowed NO extlic row -- daemon健康、值链断。\n\n**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline\nevidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller\nrestarts in /opt/xce/logs/xce_controller.log, zero extres joins after\n10:58, and esd strace showing zero network sends (debounce suppressed -- no\nrotate ever triggered again).", + "file": "src/esd/src/esd.c", + "root_cause": "esd.c's reconnect contract was \"reconnect = full re-PUSH\" (esd_on_reconnect\nsets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a\nre-JOIN: the fresh controller endpoint is created by handle_connection()\n(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets\npriv->role = HostRole_extres (join.c handle_join_). The transparent\nreconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket\nbut never re-runs the role join, and the client library's rejoin hook\n(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the\nun-joined endpoint's resval update falls through BOTH branches of\ndo_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and\nreturns rc = -EPERM == -1 forever. (The worker solved this exact shape with\nthe needs_rejoin contract, controller_connector.c; the scheduler likewise.\nThe esd -- written later -- re-used the flag machinery but dropped the\nre-join itself.)", + "fix": "src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and\ng_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing\n`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread\n(outside the library's failover lock -- the xcecd_on_reconnect no-I/O\nconstraint), and only then does the full re-push. Join failure keeps the\nflag set (next tick retries -- the worker's self-healing cadence).\n\nVerification:\n- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push\n + test_reconnect_flags_recover_from_any_previous_state\n (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED\n with the old esd.c (compile error: g_needs_rejoin did not exist -- the\n re-join state was not represented at all), GREEN with the fix: 13/13.\n- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build\n rc=0.\n- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2\n (restart controller, observe \"esd: re-joined after reconnect\" + pushes_ok\n advancing + extlic row in `resources distribution list`).", + "tags": [ + "esd", + "reconnect", + "failover" + ], + "related_bugs": [ + "bug-1760" + ], + "fix_commits": [ + "72301fd5" + ], + "record": ".wolf/bugs/bug-1978.md" } ] } diff --git a/.wolf/bugs/bug-1972.md b/.wolf/bugs/bug-1972.md new file mode 100644 index 00000000..0c5fec6a --- /dev/null +++ b/.wolf/bugs/bug-1972.md @@ -0,0 +1,86 @@ +--- +id: "bug-1972" +status: "fixed" +status_updated: "2026-10-26" +status_note: "" +component: "controller" +severity: "crash" +timestamp: "2026-10-26T09:46:58Z" +last_seen: "2026-10-26T09:46:58Z" +occurrences: 4 +files: ["src/controller/testsuite/test_job_live_gc.c"] +tags: ["test-only", "uaf", "gc"] +related: [] +fix_commits: ["f780a166"] +--- + +# bug-1972: test_job_live_gc asserted destruction by reading freed memory + +## Description +**Observed:** +`controller_unit_test` aborts under ASAN at `test_sweep_purges_expired_done_job` +(test_job_live_gc.c:169): + +``` +==27834==ERROR: AddressSanitizer: heap-use-after-free on address 0x5110000da290 +READ of size 8 at 0x5110000da290 thread T0 + #0 job_get_refcount ../lib/jobj.c:715 + #1 test_sweep_purges_expired_done_job test_job_live_gc.c:169 +freed by: + #3 job_dereference ../lib/jobj.c:705 + #5 job_live_gc_sweep ../controller/job.c:4777 +``` + +**Expected:** a correct sweep dereferences the purged job down to refcount 0 +and destroys it; the test must verify destruction without touching the freed +object. + +**Impact:** controller suite aborts (killed by SIGABRT); the whole +`controller test` meson entry fails. Test-only — the production sweep +(job.c:4765-4777, double dereference: collection ref + table's makeJob ref) +is correct. + +**Found by:** ASAN, first full-suite run after d028c8c3 (the archives' earlier +runs all predate that commit). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.39 (host ldd) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: feature/license-resources @ 68eabd8a-dirty (introduced in d028c8c3) +- Deployment: n/a (unit test) + +## Reproduction +1. `cd src/xce.build && ASAN_OPTIONS=detect_leaks=0 ./controller/controller_unit_test` +- Frequency: deterministic (4/4 runs, 2026-10-26) +- Evidence: /home/lchen/code/xce/src/testlogs.archive/xce.build/this-20261026-082318-rc1 + +## Root cause +The assertion `TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired))` runs +AFTER `job_live_gc_sweep` has already dereferenced the job to 0 and freed the +entire RTTI object (header + private + data in one calloc block). Reading the +refcount through the dangling `job_t*` is itself the use-after-free: the +stronger the sweep does its job, the more surely the test crashes. The same +pattern is absent from test_apply_pump.c's refcount test because there the +job is never destroyed. + +## Fix +Move the refcount probe BEFORE the sweep and invert its meaning: assert +refcount == 1 (table's seed reference, nothing else) before `job_live_gc_sweep`, +then verify destruction by absence — `jlg_find_job(81001) == NULL` plus the +idempotent second sweep (both already present). `expired` is never dereferenced +after the sweep; a comment marks the last safe point. + +## Verification +- Test: test_sweep_purges_expired_done_job (same file) — RED before (deterministic + ASAN abort 4/4), GREEN after (rc=0, 1354/1354 controller unity tests, zero + ASAN reports). +- Suites: controller_unit_test 1354 tests 0 failures (ASAN, detect_leaks=0); + sched_repl_trace_baseline_test 4/4 with regenerated fixtures (separate issue, + bug-free fixture staleness from fc33a5da's schema change, handled in the same + working tree). +- Live: n/a (test-only defect; production sweep unchanged). + +## References +- Related: none diff --git a/.wolf/bugs/bug-1973.md b/.wolf/bugs/bug-1973.md new file mode 100644 index 00000000..80b34298 --- /dev/null +++ b/.wolf/bugs/bug-1973.md @@ -0,0 +1,86 @@ +--- +id: "bug-1973" +status: "fixed" +status_updated: "2026-10-27" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-27T03:27:47Z" +last_seen: "2026-10-27T03:27:47Z" +occurrences: 1 +files: ["src/controller/cluster.c", "src/controller/testsuite/test_config_resource_reconcile.c"] +tags: ["config", "reconcile", "E2", "external-license"] +related: ["bug-1854"] +fix_commits: ["2fcacac2"] +summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered" +--- + +# bug-1973: cluster_build_resources aborted at first EEXIST: new resource defs never registered + +## Description +**Observed:** +Controller log on t1, after pushing a cluster.bin whose resources section ADDED +the new `extlic` definition (external-license-resources E2 Phase 1): + +``` +[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory +``` + +The xceesd daemon joined and pushed extlic values, but the controller had +no `extlic` registration at all (update_resval key_find → -ENOENT). + +**Expected:** a config activation that adds a new resource definition must +register it in the running controller's resmgr, so daemon pushes land. + +**Impact:** every newly-added resource definition was silently inert until +the controller process was restarted (boot-time registration path did run). +For E2 this meant the whole extres-daemon chain was dead on a live controller. + +**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.42 (de-nix deployment closure) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: debug (build.dbg), deployed with patchelf interpreter+RUNPATH +- Code: feature/license-resources @ ba42fdd8-dirty +- Deployment: t1 single-node (controller+scheduler+apiserver+worker), xceesd extres daemon + +## Reproduction +1. Controller running with a cluster.bin containing resources {slots, ...} +2. `xceadmin config build && xceadmin config push` with a bin that ADDS `extlic` +3. Start xceesd; it joins and pushes extlic values +4. Controller: `update_resval 'extlic': No such file or directory` (-ENOENT) +- Frequency: deterministic +- Evidence: controller log lines quoted above (t1, 2026-10-26 17:40) + +## Root cause +`cluster_build_resources()` looped over the decoded config's resource list +calling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247) +returns NULL with errno=EEXIST when the name is already registered — which is +the COMMON case at activation (boot already registered some definitions). The +old loop treated that rc as fatal and aborted the whole loop, so every +definition AFTER the first already-registered one was never registered. + +## Fix +Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under +one write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally; +RES_APPLY_CHANGED updates the registered object's definition fields +(valtype/ordtype/flags/interval) IN PLACE — none of them touches the +registration slot (idx) or accumulated accounting; mirrors +reconcile_config_queues' in-place refresh precedent (second live bug, see +bug-1974 — kept here as one loop). Negative rc still propagates. + +## Verification +- Test: `test_reconcile_resources_registers_new_defs_past_eexist` + + `test_reconcile_resources_hot_applies_changed_definition` + (src/controller/testsuite/test_config_resource_reconcile.c); RED before the + fix (lic1 unregistered / flags stayed 0x00), GREEN after. +- Suites: controller testsuite 1358 tests, 0 failures (build.dbg). +- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in + sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at + free=1; release re-admitted it. + +## References +- Related: bug-1854 (resourceApplyDef change-detection contract this loop is + built on), E2 spec .kiro/specs/external-license-resources/design.md §3.2 diff --git a/.wolf/bugs/bug-1974.md b/.wolf/bugs/bug-1974.md new file mode 100644 index 00000000..1b0eceb3 --- /dev/null +++ b/.wolf/bugs/bug-1974.md @@ -0,0 +1,85 @@ +--- +id: "bug-1974" +status: "fixed" +status_updated: "2026-10-27" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-27T03:35:12Z" +last_seen: "2026-10-27T03:35:12Z" +occurrences: 1 +files: ["src/controller/cluster.c", "src/controller/worker.c", "src/xceadmin/src/xcecommon/mapping.py"] +tags: ["config", "flags", "E2", "external-license"] +related: ["bug-1854", "bug-1973"] +fix_commits: ["2fcacac2"] +--- + +# bug-1974: config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM + +## Description +**Observed:** +Controller log on t1 after re-pushing a corrected cluster.bin (extlic mapped +with `location: [external]` → flags SHARED|GLOBAL|EXTERNAL): + +``` +[ERROR] [log_update_resval_failure] update_resval `extlic': Operation not permitted +``` + +**Expected:** update_val_res' SHARED branch routes a host=="" resval for a +SHARED-flagged resource into c->sharedinst; with correct flags the push lands. + +**Impact:** the extres daemon's pushes were permanently refused on a controller +whose registration predated the flag fix — the stale shape was frozen for the +process lifetime, so re-pushing a corrected config did not heal it. + +**Found by:** E2 Phase 1 live E2E on t1. + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.42 (de-nix deployment closure) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: debug (build.dbg) +- Code: feature/license-resources @ ba42fdd8-dirty +- Deployment: t1 single-node; xceesd extres daemon + +## Reproduction +1. Register extlic with flags=0 (see contributing factor below) +2. Push a corrected cluster.bin where extlic has SHARED|GLOBAL|EXTERNAL +3. xceesd pushes extlic=4 (host=="") +4. Controller: `update_resval 'extlic': Operation not permitted` (-EPERM) +- Frequency: deterministic (persists until controller restart or the bug-1973 fix) +- Evidence: controller log lines quoted above (t1, 2026-10-26 18:40/18:45) + +## Root cause +Two compounding factors: +1. **Producer**: the DEPLOYED xceadmin venv (/opt/xce-denix/share/xce/ + xceadmin-venv) carries a stale mapping.py with NO resourcemap handling + (83-line diff vs tree) — cluster bins it builds carry flags=0 for every + resource. Ops issue, fixed by building with the tree venv + (src/build.dbg/xceadmin/.venv/bin/xceadmin); the deployed venv still needs + redeployment. +2. **Controller**: even with a corrected bin, `cluster_build_resources()` + treated an already-registered name as "nothing to do" (bug-1854's + RES_APPLY_CHANGED was reported but never applied), so the stale flags=0 + shape survived every activation. + +## Fix +`cluster_build_resources()` now applies RES_APPLY_CHANGED by updating the +registered object's definition fields (valtype/ordtype/flags/interval) in +place under the resmgr write lock — none of them touches the registration +slot (idx) or accumulated accounting; mirrors reconcile_config_queues' +"updated by configuration activation" in-place refresh precedent. (Same loop +as bug-1973's fix.) Producer side tracked separately as ops follow-up: +redeploy the xceadmin venv. + +## Verification +- Test: `test_reconcile_resources_hot_applies_changed_definition` + (src/controller/testsuite/test_config_resource_reconcile.c); RED before + (flags stayed 0x00), GREEN after (0x1C, idx preserved). +- Suites: controller testsuite 1358 tests, 0 failures (build.dbg). +- Live: t1 — bin rebuilt with tree venv (gen=15) + fix deployed; xceesd push + landed (extlic=4 in `resources distribution list`), rusage[extlic=3] job RUN. + +## References +- Related: bug-1854 (resourceApplyDef change-detection), bug-1973 (same loop's + EEXIST abort), worker.c update_val_res guard (the -EPERM surface) diff --git a/.wolf/bugs/bug-1975.md b/.wolf/bugs/bug-1975.md new file mode 100644 index 00000000..c4800e1c --- /dev/null +++ b/.wolf/bugs/bug-1975.md @@ -0,0 +1,89 @@ +--- +id: "bug-1975" +status: "fixed" +status_updated: "2026-10-27" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-27T03:40:05Z" +last_seen: "2026-10-27T03:40:05Z" +occurrences: 1 +files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"] +tags: ["esd", "external-license", "reconnect", "E2"] +related: ["bug-1760", "bug-1973", "bug-1974"] +fix_commits: ["2fcacac2"] +summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)" +title: "xceesd idle on dead connection after controller restart (stable value: no re-push)" +--- + +# bug-1975: xceesd idle on dead connection after controller restart (stable value: no re-push) + +## Description +**Observed:** +Old xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after +the controller was restarted: + +``` +(no socket() attempts; strace showed only collector forks + epoll_wait; + /proc/net socket table had no trace of the controller connection) +``` + +Last log line was the boot-time `esd full re-push: 0/1 source(s)` — no +further activity. + +**Expected:** within one poll cycle of the controller's restart the daemon +must re-establish the connection and re-push its value (the new primary has +no record of the daemon or its value). + +**Impact:** after any controller restart/failover with a stable license +value, the controller restarts with extlic read 0 (seed skipped for EXTERNAL +by design) and no daemon re-push — every rusage[extlic] job PENDs forever +with NO_RESOURCE until the license value happens to change. + +**Found by:** E2 Phase 1 live E2E on t1 (controller restart step). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.42 (de-nix deployment closure) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: debug (build.dbg) +- Code: feature/license-resources @ ba42fdd8-dirty +- Deployment: t1 single-node; xceesd extres daemon + +## Reproduction +1. xceesd running, pushing a value the collector keeps returning unchanged +2. Restart (or fail over) the controller +3. Observe the daemon: no reconnect attempts, pushes never resume +- Frequency: deterministic whenever the value is stable across the restart +- Evidence: strace captures /tmp/esd-strace*.out (t1, 2026-10-26); log silence quoted above + +## Root cause +Three mechanisms conspired: +1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip + push) — with a stable value it pushes NOTHING. +2. A reconnect inside the client library only fires while INSIDE a push's + failover call — no pushes, no reconnect. +3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate; + with no push attempt it never runs. + +## Fix +`esd_run()`'s main loop now checks `xcecli_is_connection_broken()` +(bug-1760's pure predicate — the transport's RDHUP handler sets +XCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches. +The write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the +full re-push contract for the failover path. + +## Verification +- Test: covered by the live contract (client-side predicate already unit + tested by bug-1760's suite); esd-side loop is integration-shaped (the + trigger is a transport callback), so live E2E is the proof. +- Suites: controller testsuite 1358 tests, 0 failures (build.dbg). +- Live: t1 — controller killed and restarted; esd logged + `connection broken -- forcing a full re-push to reconnect`, rotated to the + new primary, re-pushed; `xcecli resources distribution list` shows extlic=4 + and a fresh rusage[extlic=3] job RUNS. + +## References +- Related: bug-1760 (the pure broken-connection predicate this reuses), + E2 spec design.md §3.5 E2E checklist (daemon kill/restart + controller + failover items) diff --git a/.wolf/bugs/bug-1976.md b/.wolf/bugs/bug-1976.md new file mode 100644 index 00000000..9a1e7b66 --- /dev/null +++ b/.wolf/bugs/bug-1976.md @@ -0,0 +1,90 @@ +--- +id: "bug-1976" +status: "fixed" +status_updated: "2026-10-28" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-28T05:37:09Z" +last_seen: "2026-10-28T05:26:00Z" +occurrences: 1 +files: ["src/controller/job.c", "src/controller/testsuite/test_dispatch_commit.c"] +tags: ["external-license-resources", "limits", "shared-resources", "dispatch", "e2-phase3"] +related: ["bug-1860"] +fix_commits: ["76ddd75f"] +summary: "controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced" +--- + +# bug-1976: controller dispatch commit drops allocs_shared — vres usage never charged, limits unenforced + +## Description +**Observed:** +E2 Phase 3 t1 E2E: 4 projA jobs with `-R "rusage[extlic=1]"` against a +per-project limit `extlic_share` (scope projA, cap 3.0) — ALL FOUR ran +instead of the 4th PENDing with reason limit_global. Live gdb on the +scheduler showed `sharedInUse[36/37/38]` (the vres instances) = 0.0 with +3 projA jobs RUN: the vres charge never landed. + +**Expected:** the 4th projA job PENDs (limit_global); killing a RUN job +re-admits it; projB unaffected (own per-project instance). + +**Impact:** every scheduler-side global limit (the limit vres mechanism) +was advisory only — no quota was ever enforced, for ANY shared resource +carried via allocs_shared, since dispatch-time adoption overwrote the +scheduler's own charged accounts with a shared-stripped copy. + +**Found by:** spec task (external-license-resources E2 Phase 3 E2E). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.39 +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: debug (build.dbg) +- Code: feature/license-resources @ 76ddd75f (bug present since dispatch + commit was introduced, f4b1ba4c-era) +- Deployment: t1 single-node controller + t2/t3 workers (de-nix binaries + in /opt/xce-denix), xceesd reporting extlic via fake_lic.sh + +## Reproduction +1. Push a config with a per-project limit (limits.yaml `extlic_share`, + cap 3.0) and an EXTERNAL resource extlic with ≥4 free. +2. Submit 4 jobs `-P projA -R "rusage[extlic=1]"`; observe all 4 RUN. +3. gdb scheduler: sharedInUse[vres idx] = 0.0 despite 3+ RUN jobs. +- Frequency: deterministic. +- Evidence: this record + t1 E2E transcript in session (2026-10-28). + +## Root cause +The controller's dispatch-time commit (`dispatch_a_job_()`, job.c) +rebuilt `j->job.allocs` from the scheduler's dispatch payload but +migrated ONLY the per-host records (`MOVE_PTR(j->job.allocs->allocs, +allocs)`): `nres_shared` stayed 0 and the payload's `allocs_shared` +was neither moved nor freed (a leak on top). That rebuilt record is +what `sched_job_update_fun()` re-sends to the scheduler as ground +truth, and the scheduler's JOB_UPDATE adoption (bug-1860 branch, +core_events.c) rebuilds `job->accounts` from exactly that wire copy — +so `accounts_shared == NULL`, `sched_account_jobs` charges nothing to +the vres, and `unsatisfy_` sees cap ≥ 0 + want forever. The scheduler's +own commit (`sched_commit_decision`, scheduler.c:1806-1812) built the +accounts correctly — the controller's "ground truth" clobbered it. + +## Fix +job.c dispatch-time commit: migrate the shared records alongside the +host records — `MOVE_PTR(j->job.allocs->allocs_shared, +alloc->allocs_shared); j->job.allocs->nres_shared = +alloc->nres_shared; alloc->nres_shared = 0;` (MOVE, not copy — the +caller frees the payload; also closes the leak). + +## Verification +- Test: `test_dc_success_migrates_shared_allocs` + (src/controller/testsuite/test_dispatch_commit.c). RED before the fix + ("Expected 2 Was 0" on nres_shared), GREEN after. +- Suites: full meson test 136/136 OK (build.dbg), including controller + unit suite 1361/0. +- Live: t1 E2E — 3 projA RUN / 4th PEND, projB RUN (own instance), kill + one RUN job → pending projA re-admitted, external free pool + (fake_lic) enforced independently; all daemons healthy throughout. + +## References +- Related: bug-1860 (the JOB_UPDATE ground-truth adoption that made the + stripped copy authoritative), external-license-resources design.md + §3.5/§3.6 (per-app→per-project terminology). diff --git a/.wolf/bugs/bug-1977.md b/.wolf/bugs/bug-1977.md new file mode 100644 index 00000000..90d6d57a --- /dev/null +++ b/.wolf/bugs/bug-1977.md @@ -0,0 +1,85 @@ +--- +id: "bug-1977" +status: "fixed" +status_updated: "2026-10-28" +status_note: "" +component: "controller" +severity: "crash" +timestamp: "2026-10-28T11:56:58Z" +last_seen: "2026-10-28T11:56:58Z" +occurrences: 1 +files: ["src/controller/testsuite/test_config_resource_reconcile.c"] +tags: ["testsuite", "asan"] +related: [] +fix_commits: ["8c1995c2"] +--- + +# bug-1977: crr_build_config callers' stack arrays smaller than builder -- ASAN stack overflow + +## Description +**Observed:** +``` +ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffc... +WRITE of size 8 at 0x7ffc... thread T0 + #0 crr_build_config testsuite/test_config_resource_reconcile.c:126 (dists[3] = &d4) + #1 test_reconcile_resources_registers_and_seeds_shared_pools testsuite/test_config_resource_reconcile.c:161 +``` +Every test in the file that calls `crr_build_config` crashes the same way; +the full controller suite reports 137/137 FAIL (meson reports each fixture +crash as its own failure). + +**Expected:** all four callers pass arrays large enough for the builder's +fixtures; the suite stays green. + +**Impact:** the whole controller testsuite was unrunnable under ASAN since the +external-license-resources test fixtures landed (commit 2fcacac2) -- any +regression in the suite was invisible behind the deterministic crash. + +**Found by:** ASAN -- full regression build for the license-resource feature +(`src/tools/xce-build.sh controller`, profile=asan). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: host ldd reports glibc 2.39 +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) +- Code: feature/license-resources @ 30c51ea1-dirty (introduced by 2fcacac2) +- Deployment: n/a (unit testsuite only) + +## Reproduction +1. `src/tools/xce-build.sh controller` (asan profile) -- meson test crashes: + `controller - xce:controller test` FAIL with the stack-buffer-overflow above. +- Frequency: deterministic +- Evidence: src/testlogs.archive/xce.build/this-20261028-114408-rc1/meson-logs/testlog.txt (bug-1971 failure-preservation mechanism) + +## Root cause +`crr_build_config` (test_config_resource_reconcile.c:70) registers FIVE +resources (lic1/hostres/dyn1/extlic/slots -> writes `resources[4]`) and FOUR +distributions (writes `dists[3]`) and FIVE vals (writes `vals[4]`). The four +caller test functions declared only +`resources[4] / dists[3] / vals[4]`, so the builder's writes to index 4 / 3 / +4 overflowed the callers' stack arrays. When the extlic E2 tests were added to +the builder's fixture set (2fcacac2), none of the caller declarations were +resized. (Earlier builds of this branch compiled only a worker-focused subset +-- `test_config_resource_reconcile` was not linked -- so the overflow went +unnoticed until the full controller suite was built.) + +## Fix +Resize all four caller arrays to match the builder: `resources[5]`, +`dists[4]`, `vals[5]`, each with a bug-1977 comment pointing at the sizing +contract. Files: src/controller/testsuite/test_config_resource_reconcile.c +(test_reconcile_resources_registers_and_seeds_shared_pools, +test_reconcile_resources_registers_new_defs_past_eexist, +test_reconcile_resources_hot_applies_changed_definition, +test_reconcile_resources_is_idempotent_and_hot_applies). + +## Verification +- Test: whole-file suite `controller - xce:controller test` (meson): RED + 137/137 before the fix, GREEN after. +- Suites: `src/tools/xce-build.sh controller` profile=asan: 6/6 OK + (sched push, sched worker change, apiserver push, host runusage, no-fork + rule, controller 91s). +- Live: n/a (testsuite-only defect; no production code touched). + +## References +- Related: none diff --git a/.wolf/bugs/bug-1978.md b/.wolf/bugs/bug-1978.md new file mode 100644 index 00000000..f763d290 --- /dev/null +++ b/.wolf/bugs/bug-1978.md @@ -0,0 +1,110 @@ +--- +id: "bug-1978" +status: "fixed" +status_updated: "2026-10-28" +status_note: "" +component: "controller" +severity: "wrong-result" +timestamp: "2026-10-28T12:47:59Z" +last_seen: "2026-10-28T12:47:59Z" +occurrences: 3 +files: ["src/esd/src/esd.c"] +tags: ["esd", "reconnect", "failover"] +related: ["bug-1760"] +fix_commits: ["72301fd5"] +--- + +# bug-1978: esd reconnect re-pushes without re-join -- pushes -EPERM forever + +## Description +**Observed:** +``` +t2 /tmp/xceesd-d.log: +[2026-10-28 09:29:49,019] ERROR esd push extlic=4.000000 failed (rc=-1) +[2026-10-28 09:29:49,019] INFO esd full re-push: 0/1 source(s) +[2026-10-28 10:58:22,606] ERROR esd push extlic=4.000000 failed (rc=-1) +... (every 10s poll, forever) +``` +After any controller restart the daemon's pushes fail rc=-1 (-EPERM) on +every attempt. The daemon LOOKS healthy: the TCP connection is ESTABLISHED, +the poll loop runs, the collector script fires every 10s, the status file +refreshes with `stale: false` -- only pushes_fail climbs. + +**Expected:** after the reconnect the daemon re-joins (HostRole_extres) and +pushes resume (pushes_ok advances, `xce resources distribution list` shows +extlic). + +**Impact:** the entire external-license value chain dies at the first +controller restart: extlic never reaches c->sharedinst, the scheduler never +hears a RESVALS_DELTA, every `rusage[extlic=...]` job PENDs forever +(admission sees free=0). Found during the license-resource regression: the +regression config push (gen 19) succeeded but `resources distribution list` +showed NO extlic row -- daemon健康、值链断。 + +**Found by:** regression test -- live t1/t2 E2E (2026-10-28). Timeline +evidence: pushes_ok=4 (all pre-restart), the 09:29/10:58 controller +restarts in /opt/xce/logs/xce_controller.log, zero extres joins after +10:58, and esd strace showing zero network sends (debounce suppressed -- no +rotate ever triggered again). + +## Environment +- OS: Ubuntu 24.04.2 LTS; kernel Linux 6.8.0-147-generic x86_64 +- libc: glibc 2.39 (t2 host) +- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1 +- Build: asan (xce.build) -- failing binary was the de-nix deploy + /opt/xce-denix/bin/xceesd (Oct 27 10:51, non-ASAN deploy build) +- Code: feature/license-resources @ 30c51ea1 (esd) vs controller b38ef6de +- Deployment: 3-node t1/t2/t3; xceesd on t2, controller+scheduler on t1 + +## Reproduction +1. Start xceesd on t2 (`/opt/xce-denix/bin/xceesd -c /tmp/e2e/esd.yaml ...`); + observe pushes_ok advance (the boot join works). +2. Restart the t1 controller (or kill it long enough for the connection to + drop and transparently reconnect). +3. Watch /tmp/xceesd-d.log: every subsequent `esd push ... failed (rc=-1)`. + Controller log logs no new extres join. +- Frequency: deterministic (3 sightings: 09:29, 10:58 controller restarts + + the Oct-26 original "unknown role 5" cousin) +- Evidence: t2:/tmp/xceesd-d.log, t1:/opt/xce/logs/xce_controller.log, + t2:/tmp/e2e/esd-status.json (pushes_ok frozen at 4, pushes_fail growing) + +## Root cause +esd.c's reconnect contract was "reconnect = full re-PUSH" (esd_on_reconnect +sets g_needs_full_push; the main loop re-pushes). But a re-PUSH is not a +re-JOIN: the fresh controller endpoint is created by handle_connection() +(cluster.c) with `priv->role = HostRole_client`, and only OPCode_join sets +priv->role = HostRole_extres (join.c handle_join_). The transparent +reconnect path (xce_ctx_reconnect, xcectx.c) re-establishes the TCP socket +but never re-runs the role join, and the client library's rejoin hook +(xcecli_invoke_rejoin -> esd_on_reconnect) deliberately does no I/O. So the +un-joined endpoint's resval update falls through BOTH branches of +do_update_resvals (opupd.c: role is neither extres nor worker-with-idx) and +returns rc = -EPERM == -1 forever. (The worker solved this exact shape with +the needs_rejoin contract, controller_connector.c; the scheduler likewise. +The esd -- written later -- re-used the flag machinery but dropped the +re-join itself.) + +## Fix +src/esd/src/esd.c: esd_on_reconnect now sets BOTH g_needs_rejoin and +g_needs_full_push; the main loop consumes g_needs_rejoin FIRST, re-issuing +`xcecli_join(g_esd.apictx, HostRole_extres, NULL)` on the main thread +(outside the library's failover lock -- the xcecd_on_reconnect no-I/O +constraint), and only then does the full re-push. Join failure keeps the +flag set (next tick retries -- the worker's self-healing cadence). + +## Verification +- Test: esd core test -- new tests test_reconnect_sets_both_rejoin_and_full_push + + test_reconnect_flags_recover_from_any_previous_state + (src/esd/testsuite/test_esd_core.c, white-box #include of esd.c). RED + with the old esd.c (compile error: g_needs_rejoin did not exist -- the + re-join state was not represented at all), GREEN with the fix: 13/13. +- Suites: meson test --suite esd 2/2 OK (asan, xce.build); ninja full build + rc=0. +- Live: pending -- fix binary redeploy + controller-restart drill on t1/t2 + (restart controller, observe "esd: re-joined after reconnect" + pushes_ok + advancing + extlic row in `resources distribution list`). + +## References +- Related: bug-1760 (the connection-broken predicate that makes the + debounce-stall recovery work), worker needs_rejoin precedent + (controller_connector.c), bug-1522 (register-after-start ordering). diff --git a/scripts/check-no-fork.py b/scripts/check-no-fork.py index 2a69c433..d5cf38e1 100644 --- a/scripts/check-no-fork.py +++ b/scripts/check-no-fork.py @@ -21,7 +21,8 @@ import os import re import sys -COVERED = ["src/controller", "src/scheduler", "src/lib", "src/worker"] +COVERED = ["src/controller", "src/scheduler", "src/lib", "src/worker", + "src/esd"] EXCLUDED_DIRS = {"testsuite", "test", "tests"} # Fork-family calls. uv_spawn() is libuv's fork()+exec on Linux. BANNED = re.compile( diff --git a/src/configs/limits.yaml b/src/configs/limits.yaml index 37ec6ab0..a10d16c9 100644 --- a/src/configs/limits.yaml +++ b/src/configs/limits.yaml @@ -93,3 +93,35 @@ limits: resname: mem value: 0.2 percent: true + + # external-license-resources (E2 Phase 3): strategy-layer quotas on the + # externally reported extlic pool. These ride the EXISTING limit vres + # mechanism -- no new daemon, no new admission rule -- on top of Phase + # 1's underlying-resource check (total >= uncomit + wanted). Two + # distribution shapes: + # + # 1. per-project shares: projA and projB each hold their own + # "limit=extlic_share/p=" usage instance under a 3.0 cap + # (independent accounting per project; the scope lists who may + # match at all). literal members here; named project groups are + # project-config territory (out of this spec's scope). + - name: extlic_share + projects: + members: + - projA + - projB + resname: extlic + value: 3.0 + per_project: true + + # 2. per-queue distribution: queues each get their own + # "limit=extlic_qdist/q=" instance under a 4.0 cap -- how a + # site partitions one license pool across queues. + - name: extlic_qdist + queues: + members: + - normal + - night + resname: extlic + value: 4.0 + per_queue: true diff --git a/src/configs/resources.yaml b/src/configs/resources.yaml index 360d7a8b..0c2fbe09 100644 --- a/src/configs/resources.yaml +++ b/src/configs/resources.yaml @@ -5,6 +5,7 @@ resources: type: numeric order: decrease consumable: true + builtin: true description: "job slot count (built-in)" # built-in: dynamic memory load index reported by workers @@ -12,6 +13,7 @@ resources: type: numeric interval: 60 consumable: true + builtin: true order: decrease description: "available memory in bytes (built-in dynamic load index; CLI display converts to a human-readable unit)" @@ -21,6 +23,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "1-minute load average (built-in dynamic load index)" @@ -28,6 +31,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "15-minute load average (built-in dynamic load index)" @@ -35,6 +39,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "15-second load average (built-in dynamic load index)" @@ -42,6 +47,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "cpu utilization" @@ -49,6 +55,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "page ratio" @@ -56,6 +63,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "login count" @@ -63,13 +71,15 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "idle time" - + - name: swp type: numeric interval: 60 consumable: true + builtin: true order: decrease description: "swap size" @@ -77,13 +87,15 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: increase description: "io rate" - + - name: tmp type: numeric interval: 60 consumable: true + builtin: true order: decrease description: "available tmp disk space in bytes (built-in dynamic load index; CLI display converts to a human-readable unit)" @@ -91,34 +103,42 @@ resources: # controller/worker.c add_hostattrs_resvals() - name: type type: string + builtin: true description: "host OS/arch type (built-in host attribute)" - name: model type: string + builtin: true description: "host hardware model (built-in host attribute)" - name: cpuf type: numeric + builtin: true description: "CPU speed factor (built-in host attribute)" - name: maxcpus type: numeric + builtin: true description: "max CPU count (built-in host attribute)" - name: maxmem type: numeric + builtin: true description: "max memory in bytes (built-in host attribute; CLI display converts to a human-readable unit)" - name: maxswap type: numeric + builtin: true description: "max swap in bytes (built-in host attribute; CLI display converts to a human-readable unit)" - name: maxtmp type: numeric + builtin: true description: "max tmp disk space in bytes (built-in host attribute; CLI display converts to a human-readable unit)" - name: ndisks type: numeric + builtin: true description: "disk count (built-in host attribute)" # dynamic hostbased resource, external load index @@ -269,3 +289,16 @@ resources: - name: gstring description: string resource type: string + + # external-license-resources: cluster-wide license pool reported by + # the extres daemon (xceesd) -- config declares only the name/shape, + # the free counts come from the daemon's pushes + - name: extlic + description: externally reported license pool + type: numeric + interval: 0 + consumable: true + order: decrease + resourcemap: + - location: + - external diff --git a/src/configs/schema/resources.schema.json b/src/configs/schema/resources.schema.json index b11620a4..06515aa0 100644 --- a/src/configs/schema/resources.schema.json +++ b/src/configs/schema/resources.schema.json @@ -12,6 +12,7 @@ "order": { "type": "string", "enum": ["increase", "decrease", "na"] }, "consumable": { "type": "boolean" }, "release": { "type": "boolean" }, + "builtin": { "type": "boolean" }, "description": { "type": "string" }, "resourcemap": { "type": "array", diff --git a/src/controller/cluster.c b/src/controller/cluster.c index a2810a23..0ef66c03 100644 --- a/src/controller/cluster.c +++ b/src/controller/cluster.c @@ -105,6 +105,9 @@ static void handle_endpoint_error(endpoint_t* ep) case HostRole_apiserver: handle_apiserver_leave(c, ep); break; + case HostRole_extres: + handle_extres_leave(c, ep); + break; default: ERROR("unknown role %d for endpoint %p", priv->role, ep); break; @@ -1535,21 +1538,50 @@ int cluster_build_config(cluster_t* c) int cluster_build_resources(cluster_t* c) { int i; + int rc = 0; - if (c->myconf == NULL) { + if ((c->myconf == NULL) || (c->myconf->cluster_config == NULL)) { return -EINVAL; } for (i = 0; i < c->myconf->cluster_config->nres; i++) { - xce_resource_t* res; + int arc; - res = makeResource(c->resmgr, c->myconf->cluster_config->resources[i]); - if (res == NULL) { - return -errno; + arc = resourceApplyDef(c->resmgr, + c->myconf->cluster_config->resources[i]); + if (arc == RES_APPLY_CHANGED) { + /* The registered object was created from a DIFFERENT generation + * of the definition (bug-1854's change-detection contract). + * Keeping the stale shape would freeze the boot-time behavior + * for the daemon's lifetime -- the live sighting: extlic first + * registered with flags=0 (the building tool dropped its + * resourcemap), and every later activation kept refusing the + * extres daemon's pushes with -EPERM because + * update_val_res()'s sharedinst route gates on XCE_RESF_SHARED. + * Update the definition fields IN PLACE: none of them touches + * the registration slot (idx) or accumulated accounting, and + * this mirrors reconcile_config_queues' "updated by + * configuration activation" in-place refresh precedent. */ + xce_resource_t* cur = NULL; + xce_resource_t* def = c->myconf->cluster_config->resources[i]; + + lock(c->resmgr, RTTI_LOCK_WRITE); + if (key_find(c->resmgr, (void*)def->name, (void**)&cur) > 0 + && cur != NULL) { + cur->valtype = def->valtype; + cur->ordtype = def->ordtype; + cur->flags = def->flags; + cur->interval = def->interval; + INFO("resource %s definition updated by configuration " + "activation (flags=0x%x)", def->name, def->flags); + } + unlock(c->resmgr); + } else if (arc < 0) { + rc = arc; } } - return 0; + return rc; } static void resval_obj_killer(void* arg); @@ -1598,6 +1630,15 @@ static void seed_sharedinst_from_config(cluster_t* c) || (r->flags & XCE_RESF_GLOBAL) == 0) { continue; } + /* external-license-resources: an EXTERNAL pool's values come only + * from the extres daemon's free-count pushes -- seeding a static + * config value here would mask "daemon not yet up" as a phantom + * capacity (admitting jobs against licenses that do not exist). + * Until the daemon joins, get_resvals reads 0 and jobs PEND with a + * NO_RESOURCE detail naming the resource -- the honest signal. */ + if (r->flags & XCE_RESF_EXTERNAL) { + continue; + } if (r->ndistributions < 1 || r->distributions == NULL || r->distributions[0] == NULL || r->distributions[0]->value == NULL) { continue; diff --git a/src/controller/extres.c b/src/controller/extres.c new file mode 100644 index 00000000..0b3fb82e --- /dev/null +++ b/src/controller/extres.c @@ -0,0 +1,78 @@ +/* + * extres.c — connected extres daemons (HostRole_extres joins) + * + * external-license-resources Phase 1: the external-resources daemon + * (xceesd) joins the controller with HostRole_extres and pushes + * cluster-level (SHARED) resource values through the ordinary resval + * update pipeline (opupd.c do_update_resvals). Mirrors the apiserver + * join/leave registration in apiserver.c: endpoint + name only, no + * workermgr slot — the daemon is not a schedulable host. The controller + * never pushes TO the daemon; the daemon re-pushes its full value set on + * every (re)connect, so nothing here needs replay/WAL treatment. + */ + +#include +#include +#include + +#include "cluster.h" +#include "xstring.h" +#include "xce/cluster_def.h" + +int add_extres(cluster_t* c, char* name, endpoint_t* ep) +{ + int i; + + for (i = 0; i < N_EXTRES_MAX; i++) { + if ((c->extresds[i].name != NULL) + && (strcmp(c->extresds[i].name, name) == 0)) { + /* re-join (reconnect or failover re-join): rebind in place */ + c->extresds[i].ep = ep; + return i; + } + } + + for (i = 0; i < N_EXTRES_MAX; i++) { + if (c->extresds[i].name == NULL) { + c->extresds[i].name = makeString(name); + if (c->extresds[i].name == NULL) { + return -ENOMEM; + } + c->extresds[i].ep = ep; + return i; + } + } + + ERROR("too many extres daemons (max %d); join from %s refused", + N_EXTRES_MAX, name); + return -ENOSPC; +} + +int handle_extres_leave(cluster_t* c, endpoint_t* ep) +{ + ep_private_t* priv; + int i; + + if (ep == NULL) { + return -EINVAL; + } + + priv = (ep_private_t*)ep->data; + if (priv == NULL) { + return -EINVAL; + } + + i = priv->u.idx; + if ((i < 0) || (i >= N_EXTRES_MAX)) { + SET_ERROR(EINVAL); + return -errno; + } + + /* Only clear if it is still our binding — a re-join may have rebound + * the slot to a new endpoint before this old one's close event ran. */ + if (c->extresds[i].ep == ep) { + c->extresds[i].ep = NULL; + } + + return 0; +} diff --git a/src/controller/include/cluster.h b/src/controller/include/cluster.h index e2d22724..595f943f 100644 --- a/src/controller/include/cluster.h +++ b/src/controller/include/cluster.h @@ -439,5 +439,9 @@ extern int schedadmin_signal_disconnect(void); extern int handle_apiserver_leave(cluster_t*, endpoint_t*); extern int add_apiserver(cluster_t*, char*, endpoint_t*); extern int apiserver_init(cluster_t*); +/* external-license-resources Phase 1: HostRole_extres daemon registration + * (extres.c) — endpoint + name only, no workermgr slot. */ +extern int handle_extres_leave(cluster_t*, endpoint_t*); +extern int add_extres(cluster_t*, char*, endpoint_t*); #endif diff --git a/src/controller/job.c b/src/controller/job.c index 294ea735..6b45e8b0 100644 --- a/src/controller/job.c +++ b/src/controller/job.c @@ -1936,6 +1936,19 @@ static int dispatch_a_job_(cluster_t* c, endpoint_t* ep, xce_job_alloc_t* alloc) MOVE_PTR(j->job.allocs->allocs, allocs); alloc->allocs = NULL; + /* external-license-resources Phase 3: the scheduler's dispatch + * payload also carries the shared-pool records (allocs_shared -- the + * per-project/per-queue limit vres charge among them). This rebuild + * is the record sched_job_update_fun() re-sends to the scheduler as + * ground truth (the JOB_UPDATE adoption there rebuilds job->accounts + * from it), so dropping the shared records here starves the + * scheduler's sharedInUse accounting and every global limit admits + * forever. Migrate them (MOVE, not copy -- the caller frees the + * payload) exactly like the per-host records above. */ + MOVE_PTR(j->job.allocs->allocs_shared, alloc->allocs_shared); + j->job.allocs->nres_shared = alloc->nres_shared; + alloc->nres_shared = 0; + j->job.state = JOB_STAT_RUN; /* bug-1697: start_time is now stamped HERE, synchronously alongside diff --git a/src/controller/join.c b/src/controller/join.c index f81b9d5a..fce86655 100644 --- a/src/controller/join.c +++ b/src/controller/join.c @@ -248,6 +248,43 @@ static int handle_apiserver_join(cluster_t* c, endpoint_t* ep, return 0; } +/* + * handle_extres_join: register the external-resources daemon (xceesd, + * external-license-resources Phase 1). Mirrors handle_apiserver_join's + * lightweight shape — endpoint + priv->u.idx only, no workermgr slot — and + * unlike the apiserver there is no on-join state push: the daemon is the + * pusher (it re-sends its full shared-value set on every reconnect). + */ +static int handle_extres_join(cluster_t* c, endpoint_t* ep, + xce_request_t* request) +{ + xce_reply_t reply; + xce_peerinfo_t* from = request->from; + int rc = 0; + + TRACE("%s extres daemon join, ep %p", __func__, ep); + + init_reply_by_req(&reply, request); + + rc = add_extres(c, from->name, ep); + if (rc >= 0) { + ep_private_t* priv = (ep_private_t*)ep->data; + + priv->u.idx = rc; + rc = 0; + } + + reply.status = rc; + reply.data_which = RequestReply_data_nodata; + xce_reply(ep, c->myhost, &reply); + + if (rc < 0) { + ERROR("extres daemon %s join failed, rc %d", from->name, rc); + return rc; + } + return 0; +} + static int handle_join_(cluster_t* c, endpoint_t* ep, xce_request_t* request) { ep_private_t* priv; @@ -319,6 +356,9 @@ static int handle_join_(cluster_t* c, endpoint_t* ep, xce_request_t* request) case HostRole_apiserver: rc = handle_apiserver_join(c, ep, request); break; + case HostRole_extres: + rc = handle_extres_join(c, ep, request); + break; default: rc = -E_XCE_BAD_ROLE; ERROR("unknown role %d from host %s", role, from->name); diff --git a/src/controller/meson.build b/src/controller/meson.build index 781b9ea0..e1f64c4d 100644 --- a/src/controller/meson.build +++ b/src/controller/meson.build @@ -5,6 +5,7 @@ controller_src = [ 'ctrlstatus.c', 'dataex.c', 'egress.c', + 'extres.c', 'ha.c', 'ha_convergence.c', 'job.c', @@ -90,6 +91,7 @@ controller_unit_test_src = [ 'testsuite/test_ctrl_metrics.c', 'testsuite/test_config_repl.c', 'testsuite/test_worker_join.c', + 'testsuite/test_extres_resvals.c', 'testsuite/test_ha_startup.c', 'testsuite/test_ha_election.c', 'testsuite/test_ha_elect_ctx_lifecycle.c', @@ -205,6 +207,7 @@ controller_unit_test_src = [ 'config_repl.c', 'slowlane.c', 'cluster.c', + 'extres.c', 'resync.c', ] diff --git a/src/controller/opupd.c b/src/controller/opupd.c index 6b46b03c..6c37ecd1 100644 --- a/src/controller/opupd.c +++ b/src/controller/opupd.c @@ -23,7 +23,38 @@ static int do_update_resvals(cluster_t* c, endpoint_t* ep, if (priv != NULL) { int idx = priv->u.idx; - if ((priv->role == HostRole_worker) && (idx >= 0)) { + if (priv->role == HostRole_extres) { + /* external-license-resources Phase 1: the extres daemon + * (xceesd) pushes cluster-level values with NO workermgr slot + * -- w==NULL routes every entry through update_val_res()'s + * "SHARED 即收" guard (shared values land in c->sharedinst; + * host-based values are refused -EPERM per entry). The reply + * status is 0 for any syntactically valid list: per-entry + * refusals are logged by log_update_resval_failure() and must + * not fail the whole update (mirrors the worker path's + * per-item convention). */ + xce_update_data_t* updatemsg = request->body.updatemsg; + xce_resval_list_t* resvals = updatemsg->data.resvals; + xce_resval_list_t applied = { .count = 0, .vals = NULL }; + + if ((resvals != NULL) && (resvals->count > 0)) { + applied.vals = (xce_resval_t**)calloc( + (size_t)resvals->count, sizeof(xce_resval_t*)); + } + + xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_RESOURCE, + HOOK_BEFORE_EVENT, HOOK_DETAIL_RESOURCE_UPDATE, c, + resvals); + update_resval_list(c, NULL, resvals, + (applied.vals != NULL) ? &applied : NULL); + if (applied.count > 0) { + xce_run_hooks(&(c->hooksys), XCE_CONTROLLER_HOOK_RESOURCE, + HOOK_AFTER_EVENT, HOOK_DETAIL_RESOURCE_UPDATE, c, + &applied); + } + free(applied.vals); + rc = 0; + } else if ((priv->role == HostRole_worker) && (idx >= 0)) { worker_t* w = NULL; rc = vget(c->workermgr, idx, (void**)&w); diff --git a/src/controller/testsuite/controller_test.c b/src/controller/testsuite/controller_test.c index d3cf9140..fa0284b1 100644 --- a/src/controller/testsuite/controller_test.c +++ b/src/controller/testsuite/controller_test.c @@ -40,6 +40,7 @@ extern void test_p3_manual_failover(void); extern void test_ctrl_metrics(void); extern void test_config_repl(void); extern void test_worker_join(void); +extern void test_extres_resvals(void); extern void test_ha_startup(void); extern void test_ha_election(void); extern void test_ha_elect_ctx_lifecycle(void); @@ -256,6 +257,7 @@ int main(int argc, char* argv[]) TEST(test_ctrl_metrics); TEST(test_config_repl); TEST(test_worker_join); + TEST(test_extres_resvals); TEST(test_ha_startup); TEST(test_ha_election); TEST(test_ha_elect_ctx_lifecycle); diff --git a/src/controller/testsuite/test_config_resource_reconcile.c b/src/controller/testsuite/test_config_resource_reconcile.c index 46ef051d..fd747923 100644 --- a/src/controller/testsuite/test_config_resource_reconcile.c +++ b/src/controller/testsuite/test_config_resource_reconcile.c @@ -61,13 +61,18 @@ static xce_val_t* crr_shared_val(cluster_t* c, int idx) * res0 "slots" -- built-in hostbased (no resourcemap): registered, never seeded * res1 "lic1" -- SHARED|GLOBAL, distribution value 100: seeded * res2 "hostres"-- SHARED but NOT GLOBAL (per-host shape): NOT seeded - * res3 "dyn1" -- SHARED|GLOBAL, distribution WITHOUT a value: NOT seeded */ + * res3 "dyn1" -- SHARED|GLOBAL, distribution WITHOUT a value: NOT seeded + * res4 "extlic" -- SHARED|GLOBAL|EXTERNAL, distribution value 7: NOT + * seeded (external-license-resources: an EXTERNAL pool's + * values come only from the extres daemon's free-count + * pushes; seeding a static config value would mask "daemon + * not yet up" as a phantom capacity) */ static void crr_build_config(cluster_config_t* cfg, xce_resource_t** resources, res_distribution_t** dists, xce_val_t** vals) { - static xce_resource_t r0, r1, r2, r3; - static res_distribution_t d1, d2, d3; - static xce_val_t v1, v3; + static xce_resource_t r0, r1, r2, r3, r4; + static res_distribution_t d1, d2, d3, d4; + static xce_val_t v1, v3, v4; memset(&r0, 0, sizeof(r0)); r0.name = (char*)"slots"; @@ -108,11 +113,25 @@ static void crr_build_config(cluster_config_t* cfg, xce_resource_t** resources, dists[2] = &d3; vals[3] = NULL; + memset(&r4, 0, sizeof(r4)); + r4.name = (char*)"extlic"; + r4.flags = XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL; + memset(&d4, 0, sizeof(d4)); + memset(&v4, 0, sizeof(v4)); + v4.which = Value_u_fvalue; + v4.u.fvalue = 7.0; + d4.value = &v4; + r4.ndistributions = 1; + r4.distributions = (res_distribution_t**)&dists[3]; + dists[3] = &d4; + vals[4] = &v4; + resources[0] = &r0; resources[1] = &r1; resources[2] = &r2; resources[3] = &r3; - cfg->nres = 4; + resources[4] = &r4; + cfg->nres = 5; cfg->resources = resources; } @@ -123,9 +142,13 @@ static void test_reconcile_resources_registers_and_seeds_shared_pools(void) cluster_t c; myconf_t conf; cluster_config_t cfg; - xce_resource_t* resources[4]; - res_distribution_t* dists[3]; - xce_val_t* vals[4]; + /* bug-1977: the builder registers FIVE resources (indices 0..4) and + * FOUR distributions (indices 0..3) -- the caller arrays must match, + * or the builder's resources[4]/dists[3]/vals[4] writes overflow these + * stacks (ASAN stack-buffer-overflow, deterministic). */ + xce_resource_t* resources[5]; + res_distribution_t* dists[4]; + xce_val_t* vals[5]; xce_resource_t* r; if (makeString((char*)"crr-probe") == NULL) @@ -181,6 +204,157 @@ static void test_reconcile_resources_registers_and_seeds_shared_pools(void) xce_resource_t* rb = crr_find(&c, "slots"); TEST_ASSERT_NULL(crr_shared_val(&c, rb->idx)); + /* EXTERNAL: daemon-reported free values only -- never seeded from the + * config, even with a valued distribution (external-license-resources). */ + xce_resource_t* re = crr_find(&c, "extlic"); + TEST_ASSERT_NOT_NULL(re); + TEST_ASSERT_NULL_MESSAGE(crr_shared_val(&c, re->idx), + "an EXTERNAL shared pool must not seed from its config value -- " + "only the extres daemon reports its free counts"); + + xce_hook_fini(&c.hooksys); + dereference(c.resmgr); + zarray_fini(&c.sharedinst); +} + +/* Live-activation partial registration: the resmgr already holds SOME of the + * config's resources (registered at boot from the previous generation) but + * not all (the activation adds a NEW definition -- the external-license- + * resources scenario: gen-12 config without extlic, gen-13 activation adds + * it while the 33 boot resources EEXIST). makeResource() refuses an existing + * name with NULL+EEXIST, so cluster_build_resources() must NOT abort the + * loop at the first EEXIST -- the new definition must still register and + * its shared pool (lic1 here) must still seed. Regression for the live + * sighting: every activation after boot bailed at "slots" (i=0) so NO new + * resource definition ever reached a running controller. */ +static void test_reconcile_resources_registers_new_defs_past_eexist(void) +{ + kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING, + .nslots = 16, .delta = 8 }; + cluster_t c; + myconf_t conf; + cluster_config_t cfg; + /* bug-1977: the builder writes resources[0..4]/dists[0..3]/vals[0..4] -- + * caller arrays must match its sizing. */ + xce_resource_t* resources[5]; + res_distribution_t* dists[4]; + xce_val_t* vals[5]; + xce_resource_t* r; + xce_val_t* v; + + if (makeString((char*)"crr-probe") == NULL) + xstring_table_init(64); + + memset(&c, 0, sizeof(c)); + memset(&conf, 0, sizeof(conf)); + TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX)); + c.resmgr = makeInstance(&xkvmgr_type, &str_params); + TEST_ASSERT_NOT_NULL(c.resmgr); + zarray_init(&c.sharedinst, 16, + (zarray_item_killer_t)crr_val_killer); + + crr_build_config(&cfg, resources, dists, vals); + conf.cluster_config = &cfg; + c.myconf = &conf; + + /* Boot path: register ONLY the first definition (slots) -- a smaller + * previous generation. */ + { + xce_resource_t* boot = makeResource(c.resmgr, resources[0]); + TEST_ASSERT_NOT_NULL(boot); + } + + /* Activation: the full new config (slots EEXISTS, lic1 is new). */ + TEST_ASSERT_EQUAL_INT_MESSAGE(0, reconcile_config_resources(&c), + "an activation must tolerate EEXIST on pre-registered resources " + "and keep registering the rest"); + + r = crr_find(&c, "lic1"); + TEST_ASSERT_NOT_NULL_MESSAGE(r, + "the NEW definition must register even though an earlier resource " + "in the config already existed"); + v = crr_shared_val(&c, r->idx); + TEST_ASSERT_NOT_NULL_MESSAGE(v, + "the new definition's static shared pool must seed on activation"); + TEST_ASSERT_EQUAL_FLOAT(100.0, v->u.fvalue); + + /* the pre-existing runtime object must be kept (idempotent pass) */ + { + xce_resource_t* slots = crr_find(&c, "slots"); + + TEST_ASSERT_NOT_NULL(slots); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, slots->idx, + "the boot-registered object keeps its resmgr slot"); + } + + xce_hook_fini(&c.hooksys); + dereference(c.resmgr); + zarray_fini(&c.sharedinst); +} + +/* Hot-apply a CHANGED definition: the resmgr already holds a resource with + * one definition (registered at boot from the previous generation), and the + * activation brings a CHANGED one (flags changed -- the external-license- + * resources live sighting: extlic was first registered from a config built + * WITHOUT the `location: [external]` resourcemap, so its flags were 0; the + * corrected config carries SHARED|GLOBAL|EXTERNAL). Keeping the stale object + * would leave the daemon's pushes refused -EPERM forever (update_val_res + * gates the sharedinst route on XCE_RESF_SHARED) until a manual controller + * restart. The activation must update the definition fields in place -- + * idx (runtime state) preserved -- mirroring reconcile_config_queues' + * in-place refresh contract. */ +static void test_reconcile_resources_hot_applies_changed_definition(void) +{ + kvmgr_params_t str_params = { .key_type = KVMGR_KEY_STRING, + .nslots = 16, .delta = 8 }; + cluster_t c; + myconf_t conf; + cluster_config_t cfg; + /* bug-1977: the builder writes resources[0..4]/dists[0..3]/vals[0..4] -- + * caller arrays must match its sizing. */ + xce_resource_t* resources[5]; + res_distribution_t* dists[4]; + xce_val_t* vals[5]; + xce_resource_t* r; + static xce_resource_t old_extlic; + + if (makeString((char*)"crr-probe") == NULL) + xstring_table_init(64); + + memset(&c, 0, sizeof(c)); + memset(&conf, 0, sizeof(conf)); + TEST_ASSERT_EQUAL_INT(0, xce_hook_init(&c.hooksys, XCE_CONTROLLER_HOOK_MAX)); + c.resmgr = makeInstance(&xkvmgr_type, &str_params); + TEST_ASSERT_NOT_NULL(c.resmgr); + zarray_init(&c.sharedinst, 16, + (zarray_item_killer_t)crr_val_killer); + + /* Boot path: register "extlic" WITHOUT the shared/external flags -- the + * stale gen-13 shape (resourcemap dropped by the building tool). */ + memset(&old_extlic, 0, sizeof(old_extlic)); + old_extlic.name = (char*)"extlic"; + old_extlic.flags = 0; + TEST_ASSERT_NOT_NULL(makeResource(c.resmgr, &old_extlic)); + + /* Activation: the corrected config where extlic is + * SHARED|GLOBAL|EXTERNAL (r4 in the shared builder). */ + crr_build_config(&cfg, resources, dists, vals); + conf.cluster_config = &cfg; + c.myconf = &conf; + + TEST_ASSERT_EQUAL_INT_MESSAGE(0, reconcile_config_resources(&c), + "an activation over a stale definition must succeed"); + + r = crr_find(&c, "extlic"); + TEST_ASSERT_NOT_NULL(r); + TEST_ASSERT_EQUAL_HEX32_MESSAGE( + XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL, r->flags, + "the activation must hot-apply the changed flags in place -- a " + "stale flags=0 object would refuse the daemon's pushes forever"); + TEST_ASSERT_EQUAL_INT_MESSAGE(0, r->idx, + "the runtime idx (registration slot) must be preserved by the " + "in-place update"); + xce_hook_fini(&c.hooksys); dereference(c.resmgr); zarray_fini(&c.sharedinst); @@ -197,9 +371,11 @@ static void test_reconcile_resources_is_idempotent_and_hot_applies(void) cluster_t c; myconf_t conf; cluster_config_t cfg; - xce_resource_t* resources[4]; - res_distribution_t* dists[3]; - xce_val_t* vals[4]; + /* bug-1977: the builder writes resources[0..4]/dists[0..3]/vals[0..4] -- + * caller arrays must match its sizing. */ + xce_resource_t* resources[5]; + res_distribution_t* dists[4]; + xce_val_t* vals[5]; xce_resource_t* r; xce_val_t* v; @@ -246,5 +422,7 @@ void test_config_resource_reconcile(void) { UnitySetTestFile(__FILE__); RUN_TEST(test_reconcile_resources_registers_and_seeds_shared_pools); + RUN_TEST(test_reconcile_resources_registers_new_defs_past_eexist); + RUN_TEST(test_reconcile_resources_hot_applies_changed_definition); RUN_TEST(test_reconcile_resources_is_idempotent_and_hot_applies); } diff --git a/src/controller/testsuite/test_dispatch_commit.c b/src/controller/testsuite/test_dispatch_commit.c index c50650f9..58c2de3e 100644 --- a/src/controller/testsuite/test_dispatch_commit.c +++ b/src/controller/testsuite/test_dispatch_commit.c @@ -393,6 +393,78 @@ static void test_dc_success_notifies_job_update_hook_exactly_once(void) printf("test_dc_success_notifies_job_update_hook_exactly_once: PASS\n"); } +#define DC_JOBID_SHARED 91006ULL + +/* external-license-resources Phase 3 (vres charge) RED/GREEN proof: the + * scheduler's dispatch payload (sched_finalize_job_alloc_, alloc_runner.c) + * carries BOTH the per-host records AND the shared-pool records + * (allocs_shared -- the vres instance "limit=extlic_share/p=projA" charge + * among them). The controller's dispatch-time commit (job.c) rebuilds + * j->job.allocs from that payload, and that rebuilt record is what + * sched_job_update_fun() re-sends to the scheduler as ground truth + * (bug-1860 adoption in the scheduler's JOB_UPDATE handler rebuilds + * job->accounts from it) -- so if the rebuild drops allocs_shared, the + * scheduler's accounts lose every shared record, sharedInUse never + * accumulates the vres charge, and unsatisfy_ admits past the limit + * forever. + * + * RED against the pre-fix code: j->job.allocs->nres_shared stays 0 and + * allocs_shared stays NULL (the rebuild MOVE_PTRs only allocs->allocs). + * GREEN after the fix: the shared records are migrated intact (count, + * names, values), and the payload's own shared pointer is moved out (no + * double free). */ +static void test_dc_success_migrates_shared_allocs(void) +{ + job_t* j; + xce_job_alloc_t* alloc; + xce_alloc_t** shared; + int rc; + + dc_setup(); + + j = dc_seed_ready(DC_JOBID_SHARED, 0, "dq", "wkr-dc-shared"); + + alloc = make_job_alloc(DC_JOBID_SHARED, 1, + make_host_allocs(1, + make_host_alloc((char*)"wkr-dc-shared", 1, + make_allocs(1, + make_alloc((char*)"slots", NULL, 0, 1.0f, 0))))); + TEST_ASSERT_NOT_NULL(alloc); + + shared = make_allocs(2, + make_alloc((char*)"limit=extlic_share/p=projA", NULL, 0, 1.0f, 0), + make_alloc((char*)"extlic", NULL, 0, 1.0f, 0)); + TEST_ASSERT_NOT_NULL(shared); + alloc->allocs_shared = shared; + alloc->nres_shared = 2; + + rc = dispatch_a_job_test(&g_dc, NULL, alloc); + TEST_ASSERT_TRUE_MESSAGE(rc >= 0, + "a dispatch to a live, no-iomgr worker endpoint must succeed"); + + TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, j->job.state); + TEST_ASSERT_NOT_NULL(j->job.allocs); + TEST_ASSERT_EQUAL_INT_MESSAGE(2, j->job.allocs->nres_shared, + "the dispatch-time rebuild must migrate the payload's shared-pool " + "records -- dropping them starves the scheduler's vres usage " + "accounting (sharedInUse) and defeats every global limit"); + TEST_ASSERT_NOT_NULL(j->job.allocs->allocs_shared); + TEST_ASSERT_EQUAL_STRING("limit=extlic_share/p=projA", + j->job.allocs->allocs_shared[0]->resname); + TEST_ASSERT_EQUAL_FLOAT(1.0f, j->job.allocs->allocs_shared[0]->value); + TEST_ASSERT_EQUAL_STRING("extlic", + j->job.allocs->allocs_shared[1]->resname); + TEST_ASSERT_EQUAL_FLOAT(1.0f, j->job.allocs->allocs_shared[1]->value); + TEST_ASSERT_NULL_MESSAGE(alloc->allocs_shared, + "the shared records must be MOVE_PTR'd out of the payload, not " + "copied -- a copy would double-free when the caller frees it"); + + free_JobAlloc_ptr(&alloc); + dc_drain_ep(&g_dc_worker_ep); + dc_teardown(); + printf("test_dc_success_migrates_shared_allocs: PASS\n"); +} + #define DC_JOBID_NOTFOUND 91002ULL /* Regression safety net (untouched failure branch): a dispatch whose @@ -801,6 +873,7 @@ void test_dispatch_commit(void) RUN_TEST(test_dc_success_commits_alloc_state_and_resets_counter); RUN_TEST(test_dc_success_stamps_start_time_synchronously); RUN_TEST(test_dc_success_notifies_job_update_hook_exactly_once); + RUN_TEST(test_dc_success_migrates_shared_allocs); RUN_TEST(test_dc_worker_not_found_leaves_job_pend_and_alloc_uncommitted); RUN_TEST(test_dc_success_appends_wal_job_dispatch_durably); } diff --git a/src/controller/testsuite/test_extres_resvals.c b/src/controller/testsuite/test_extres_resvals.c new file mode 100644 index 00000000..980190e8 --- /dev/null +++ b/src/controller/testsuite/test_extres_resvals.c @@ -0,0 +1,297 @@ +/* + * test_extres_resvals.c — external-license-resources Phase 1 (E2), steps 1-2 + * + * The extres daemon (xceesd) pushes cluster-level (SHARED) resource values + * through the same resval update pipeline workers use, but it has NO + * workermgr slot: it joins as HostRole_extres and updates land on + * c->sharedinst directly. Two rules pin the daemon path: + * + * 1. update_resval_list() accepts w==NULL and applies SHARED-resource + * values into c->sharedinst (the only array the daemon can write). + * 2. The "SHARED 即收" guard: a NULL worker may only write SHARED + * resources — a non-SHARED (host-based) resource with w==NULL is + * refused (-EPERM) and NOT counted as applied (bug-1583's applied-list + * contract: downstream hooks must not hear about rejected values). + * + * This pins the opupd-side admission shape BEFORE the do_update_resvals + * role gate is extended to HostRole_extres (opupd itself needs a live + * endpoint, so the seam under test is worker.c's list applier, the same + * XCE_TESTING-exposure convention as apply_join_state in + * test_worker_join.c). + * + * external-license-resources Phase 2 adds rule 3: an EXTERNAL shared + * acceptance also stamps last_report on the myconf cluster-config tree's + * definition -- the apiserver's resource-definition store is fed ONLY by + * the join-time config push (myconf tree, not the live resmgr object), so + * without this the CLI's LAST_UPDATED column would stay "never" forever. + */ + +/* rtti.h must come first — it defines 'var' and 'typeinfo_t' used by kvmgr.h */ +#include + +#include +#include +#include +#include + +#include + +#include "cluster.h" +#include "controller.capnp.h" +#include "kvmgr.h" +#include "myconf.h" +#include "resobj.h" +#include "worker.h" +#include "xce/types.h" +#include "zarray.h" + +/* ── Forward declarations for functions under test ──────────────────── */ + +/* + * XCE_TESTING exposure in worker.c (same convention as apply_join_state): + * the daemon path calls update_resval_list() with w==NULL. + */ +int update_resval_list(cluster_t* c, worker_t* w, xce_resval_list_t* l, + xce_resval_list_t* applied); + +/* ── fixture ────────────────────────────────────────────────────────── */ + +typedef struct { + cluster_t c; + var resmgr; + /* myconf fixture: one cluster_config_t owning a static resources[] tree + * (the config-push wire source, distinct from the resmgr objects). */ + myconf_t myconf; + cluster_config_t ccfg; + xce_resource_t* cfg_res[2]; + xce_resource_t cfg_defs[2]; +} test_cluster_t; + +static test_cluster_t g_te; + +static void setup_resmgr(void) +{ + kvmgr_params_t str_params = { + .key_type = KVMGR_KEY_STRING, + .nslots = 64, + .delta = 16, + .datakiller = NULL, + }; + + memset(&g_te, 0, sizeof(g_te)); + + g_te.resmgr = makeInstance(&xkvmgr_type, &str_params); + TEST_ASSERT_NOT_NULL(g_te.resmgr); + + /* one SHARED resource (daemon-writable) and one host-based resource */ + { + static const char* names[2] = { "lic", "slots" }; + static const int shared[2] = { 1, 0 }; + xce_resource_t rdef; + int i; + + for (i = 0; i < 2; i++) { + memset(&rdef, 0, sizeof(rdef)); + rdef.name = (char*)names[i]; + rdef.valtype = XCE_VAL_NUMERIC; + rdef.flags = shared[i] ? XCE_RESF_SHARED : 0; + rdef.idx = -1; + TEST_ASSERT_NOT_NULL_MESSAGE(makeResource(g_te.resmgr, &rdef), + names[i]); + } + } + + g_te.c.resmgr = g_te.resmgr; + zarray_init(&g_te.c.sharedinst, 64, NULL); + + /* the myconf tree mirrors the resmgr registrations by name (the live + * shape: cluster_build_resources applies the config tree INTO resmgr, + * so names match; objects are distinct allocations) */ + { + static const char* names[2] = { "lic", "slots" }; + + for (int i = 0; i < 2; i++) { + memset(&g_te.cfg_defs[i], 0, sizeof(g_te.cfg_defs[i])); + g_te.cfg_defs[i].name = (char*)names[i]; + g_te.cfg_res[i] = &g_te.cfg_defs[i]; + } + g_te.ccfg.nres = 2; + g_te.ccfg.resources = g_te.cfg_res; + g_te.myconf.cluster_config = &g_te.ccfg; + g_te.c.myconf = &g_te.myconf; + } +} + +static void teardown_resmgr(void) +{ + zarray_fini(&g_te.c.sharedinst); + if (g_te.resmgr) dereference(g_te.resmgr); + memset(&g_te, 0, sizeof(g_te)); +} + +/* Build a one-entry resval list for `resname` with fvalue `value`. The + * xce_val_t and the list are statics so no teardown is needed (makeResource + * interns names; the resval name pointer must outlive the call). */ +static xce_resval_list_t g_te_list; +static xce_resval_t* g_te_vals[4]; +static xce_resval_t g_te_rv[4]; +static xce_val_t g_te_val[4]; + +static xce_resval_list_t* make_resval_list(const char** names, + const float* values, int n) +{ + int i; + + for (i = 0; i < n; i++) { + memset(&g_te_val[i], 0, sizeof(g_te_val[i])); + g_te_val[i].which = Value_u_fvalue; + g_te_val[i].u.fvalue = values[i]; + + memset(&g_te_rv[i], 0, sizeof(g_te_rv[i])); + g_te_rv[i].name = (char*)names[i]; + g_te_rv[i].host = NULL; + g_te_rv[i].val = &g_te_val[i]; + + g_te_vals[i] = &g_te_rv[i]; + } + g_te_list.count = n; + g_te_list.vals = g_te_vals; + return &g_te_list; +} + +/* ── tests ──────────────────────────────────────────────────────────── */ + +/* Rule 1: daemon (w==NULL) SHARED write lands in c->sharedinst at the + * resource's idx. */ +static void test_extres_shared_resval_lands_in_sharedinst(void) +{ + static const char* names[1] = { "lic" }; + static const float values[1] = { 4.0f }; + xce_resval_list_t* l; + xce_val_t* stored; + int rc; + + setup_resmgr(); + l = make_resval_list(names, values, 1); + rc = update_resval_list(&g_te.c, NULL, l, NULL); + TEST_ASSERT_EQUAL_INT(0, rc); + + stored = (xce_val_t*)zarray_get(&g_te.c.sharedinst, 0); + TEST_ASSERT_NOT_NULL(stored); + TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which); + TEST_ASSERT_EQUAL_FLOAT(4.0, stored->u.fvalue); + teardown_resmgr(); +} + +/* Rule 2 ("SHARED 即收"): w==NULL + non-SHARED resource is refused + * (-EPERM per-item, logged by the caller's failure hook), not applied, and + * not reported as applied -- the list-level call still succeeds, matching + * update_resvals()'s per-item convention. */ +static void test_extres_nonshared_resval_refused_eperm(void) +{ + static const char* names[2] = { "slots", "lic" }; + static const float values[2] = { 8.0f, 4.0f }; + xce_resval_list_t* l; + xce_resval_list_t applied = { .count = 0, .vals = NULL }; + xce_resval_t** applied_vals; + xce_val_t* stored; + int rc; + + setup_resmgr(); + applied_vals = (xce_resval_t**)calloc((size_t)2, sizeof(xce_resval_t*)); + TEST_ASSERT_NOT_NULL(applied_vals); + applied.vals = applied_vals; + + l = make_resval_list(names, values, 2); + rc = update_resval_list(&g_te.c, NULL, l, &applied); + TEST_ASSERT_EQUAL_INT(0, rc); /* list-level call still succeeds */ + + /* the SHARED entry was applied and reported... */ + stored = (xce_val_t*)zarray_get(&g_te.c.sharedinst, 0); + TEST_ASSERT_NOT_NULL(stored); + TEST_ASSERT_EQUAL_FLOAT(4.0, stored->u.fvalue); + TEST_ASSERT_EQUAL_INT(1, applied.count); + TEST_ASSERT_EQUAL_STRING("lic", applied.vals[0]->name); + + /* ...the non-SHARED entry ("slots", first in the list) was refused and + * NOT reported as applied */ + TEST_ASSERT_EQUAL_STRING("slots", l->vals[0]->name); /* sanity */ + TEST_ASSERT_EQUAL_INT(1, applied.count); + + teardown_resmgr(); + free(applied_vals); +} + +/* Rule 3 (Phase 2): an EXTERNAL shared acceptance stamps last_report on the + * myconf cluster-config tree's definition with the current time — the + * config push the apiserver receives on (re)join serializes THAT tree, not + * the live resmgr object, so an unstamped tree would show "never" in the + * CLI's LAST_UPDATED column even while the daemon pushes normally. */ +static void test_extres_external_accept_stamps_config_tree(void) +{ + static const char* names[1] = { "lic" }; + static const float values[1] = { 4.0f }; + xce_resval_list_t* l; + time_t before; + time_t after; + + setup_resmgr(); + + /* mark the definition EXTERNAL and confirm both copies start unstamped + * (config decode leaves last_report 0 for YAML-sourced definitions) */ + { + xce_resource_t* live = NULL; + + lock(g_te.resmgr, RTTI_LOCK_READ); + (void)key_find(g_te.resmgr, (void*)"lic", (void**)&live); + unlock(g_te.resmgr); + TEST_ASSERT_NOT_NULL(live); + live->flags |= XCE_RESF_EXTERNAL; + g_te.cfg_defs[0].flags |= XCE_RESF_EXTERNAL; + + TEST_ASSERT_EQUAL_INT(0, live->last_report); + TEST_ASSERT_EQUAL_INT(0, g_te.cfg_defs[0].last_report); + } + + time(&before); + l = make_resval_list(names, values, 1); + TEST_ASSERT_EQUAL_INT(0, update_resval_list(&g_te.c, NULL, l, NULL)); + time(&after); + + TEST_ASSERT_EQUAL_INT((int)before, (int)g_te.cfg_defs[0].last_report); + TEST_ASSERT_TRUE(g_te.cfg_defs[0].last_report >= before); + TEST_ASSERT_TRUE(g_te.cfg_defs[0].last_report <= after); + + teardown_resmgr(); +} + +/* The stamp is EXTERNAL-gated: a plain SHARED (non-EXTERNAL) acceptance + * leaves the config tree untouched — last_report means "external value + * last accepted", and worker-sourced values must never set it. */ +static void test_extres_plain_shared_accept_leaves_config_tree(void) +{ + static const char* names[1] = { "lic" }; + static const float values[1] = { 4.0f }; + xce_resval_list_t* l; + + setup_resmgr(); + + /* SHARED but NOT EXTERNAL */ + l = make_resval_list(names, values, 1); + TEST_ASSERT_EQUAL_INT(0, update_resval_list(&g_te.c, NULL, l, NULL)); + + TEST_ASSERT_EQUAL_INT(0, g_te.cfg_defs[0].last_report); + + teardown_resmgr(); +} + +/* ── suite entry point ─────────────────────────────────────────────── */ + +void test_extres_resvals(void) +{ + UnitySetTestFile(__FILE__); + RUN_TEST(test_extres_shared_resval_lands_in_sharedinst); + RUN_TEST(test_extres_nonshared_resval_refused_eperm); + RUN_TEST(test_extres_external_accept_stamps_config_tree); + RUN_TEST(test_extres_plain_shared_accept_leaves_config_tree); +} diff --git a/src/controller/testsuite/test_job_live_gc.c b/src/controller/testsuite/test_job_live_gc.c index 2a820774..03bef978 100644 --- a/src/controller/testsuite/test_job_live_gc.c +++ b/src/controller/testsuite/test_job_live_gc.c @@ -160,13 +160,21 @@ static void test_sweep_purges_expired_done_job(void) TEST_ASSERT_EQUAL_INT(3, jlg_job_count()); + /* Refcount before the sweep: seed gives the table 1, and nothing else + * holds the job. Read NOW — a correct sweep dereferences down to + * job_destroy and FREES the whole RTTI object, so touching `expired` + * after the sweep is a use-after-free (the assertion that proved + * "refcount reached 0" by reading freed memory was itself the bug; + * destruction is verified by absence below instead). */ + TEST_ASSERT_EQUAL_UINT32(1, job_get_refcount(expired)); + job_live_gc_sweep(&g_jlg_c); - /* The expired job is gone from the table... */ + /* The expired job is gone from the table — and, with the fixture + * holding no other reference, a correct sweep also destroyed it + * (refcount 1 → 0 through its two dereferences). `expired` must not + * be dereferenced past this point. */ TEST_ASSERT_NULL(jlg_find_job(81001)); - /* ...and its refcount ran down to destruction (seed refcount 1, sweep - * added 1, dereference after unlink leaves 0 → job_destroy ran). */ - TEST_ASSERT_EQUAL_UINT32(0, job_get_refcount(expired)); /* The others are untouched. */ TEST_ASSERT_NOT_NULL(jlg_find_job(81002)); diff --git a/src/controller/worker.c b/src/controller/worker.c index 26d84252..32cb3aca 100644 --- a/src/controller/worker.c +++ b/src/controller/worker.c @@ -74,6 +74,42 @@ static int update_val_res(cluster_t* c, worker_t* w, xce_resource_t* r, if (r->flags & XCE_RESF_SHARED) { /* update shared resource */ array = &(c->sharedinst); + /* external-license-resources Phase 2: the descriptor's reserved + * last_report becomes the "external value freshness" stamp -- the + * extres daemon's accepted pushes are the only writers of an + * EXTERNAL shared value, so every acceptance refreshes it and + * `xce resources` / a stale read can tell "reported recently" + * from "kept old value" (design.md 3.5 运维面). The apiserver's + * definition store is fed by the join-time CONFIG push, which + * serializes c->myconf->cluster_config -- a DIFFERENT copy from + * this live resmgr object -- so that tree's same-named definition + * gets the same stamp or the CLI's LAST_UPDATED column would stay + * "never" between apiserver (re)joins. */ + if (r->flags & XCE_RESF_EXTERNAL) { + time_t now = time(NULL); + + r->last_report = now; + if ((c->myconf != NULL) && (c->myconf->cluster_config != NULL)) { + cluster_config_t* cc = c->myconf->cluster_config; + + for (int i = 0; i < cc->nres; i++) { + xce_resource_t* cr = cc->resources[i]; + + if ((cr != NULL) && (cr->name != NULL) + && (strcmp(cr->name, r->name) == 0)) { + cr->last_report = now; + break; + } + } + } + } + } else if (w == NULL) { + /* external-license-resources Phase 1: a NULL worker is the extres + * daemon (HostRole_extres join, no workermgr slot). It may only + * write SHARED (cluster-level) resources -- a host-based resource + * has no per-host storage without a worker, and the daemon must + * never fabricate host values. Refuse loudly. */ + return -EPERM; } else { /* update host based resource */ array = &(w->host.resvals); diff --git a/src/docs/xce.md b/src/docs/xce.md index 18b3b5d0..e8f292ee 100644 --- a/src/docs/xce.md +++ b/src/docs/xce.md @@ -122,8 +122,9 @@ supports a subset of the LSF array syntax. - **Element number in the job's environment.** A running element finds its own number in `XCE_JOBINDEX` (1..N). An ordinary job gets `XCE_JOBINDEX=0`. A user-supplied `XCE_JOBINDEX` is overridden, with a - warning in the worker log. `XCE_JOBID` is the element's own job number, as - before. + warning in the worker log. `XCE_JOBID` is the element's own job number in + its written form (`cluster-apiserver-timestamp-sequence`, the same form + listings and control commands accept). - **Test harness.** `xce_test jobsub -a N` submits elements 1 through N (2 <= N <= 32767, regardless of `max_array_size`; `-a 1` submits an ordinary job). `$jobid` is then the array identity and `$jobid[k]` names @@ -156,6 +157,48 @@ These LSF features are not available either: - **`LSB_JOBINDEX` / `LSB_JOBID` variables** are not set. Scripts read `XCE_JOBINDEX` (element number) and `XCE_JOBID` (job number) instead. +## Resource requirement (resreq) + +A job's resource request is given at submission with +`xcecli jobs submit -R ""` (REST: `resreq`). The grammar accepts +`select[...]`, `rusage[...]`, `span[...]`, `same[...]` and `affinity[...]` +clauses in any order, compound requests (`N*{...} + M*{...}`) and +alternatives (`{...} || {...}`). A job with no request (or an empty one) +is scheduled as `type == any` (any host) — the API layer does not apply a +default; the "default: type == local" note in the API model is historical. + +What the scheduler actually enforces today: + +- **`select[]`** — enforced. The expression is evaluated + against each candidate host's registered resources (`resources.yaml`: + `slots`, `mem`, load indexes, host attributes `type`/`model`/... and any + static resources). Workers report `type` as `LINUX64` and `model` as + `Intel64`. + - An unknown resource name in the expression is **tolerated, not + rejected**: the select is dropped and the job is scheduled without it, + with a warning in the scheduler log. Everything after the dropped + select in the same request (e.g. a trailing `rusage[...]`) is dropped + with it. A malformed select (syntax error, type mismatch) rejects the + whole request at submission or scheduling. +- **`rusage[=[:duration=[:decay=<0|1>]] ...]`** — the + value is enforced as a per-task request (multiplied by the job's slot + count; `mem` also feeds memory limits and accounting). The optional + `per` suffix (`value/task`, `value/job`, `value/host`) is parsed and + kept, and `value/job` / `value/host` are honored when computing + job-wide totals, but the allocation request itself is always built + **per task**. `duration` and `decay` are parsed and stored but **not + consumed** — no historical-usage decay is implemented (see gaps.md). +- **`span[...]`, `same[...]`, `affinity[...]`** — parsed for syntax and + **dropped**: a multi-host job may be split across hosts by even + splitting only, with no placement control (see gaps.md #15). +- **Queue-level `res_req` / `gpu_req`** (`queues.yaml`) — the queue's + `res_req` string is carried to the scheduler's queue object but is + **never merged into job allocation**; jobs get only their own request. + `gpu_req` has no wire field at all. +- **Ordering** — requests are evaluated first-fit in submission order + within a queue; there is no `order[...]` clause and no load-based + ordering. + ## shell An interactive bash shell used for troubleshooting and inspection. Not diff --git a/src/esd/include/esd.h b/src/esd/include/esd.h new file mode 100644 index 00000000..15894cd1 --- /dev/null +++ b/src/esd/include/esd.h @@ -0,0 +1,130 @@ +#if !defined(_ESD_H_) + +#define _ESD_H_ 1 + +#include +#include +#include + +#include "config.h" +#include "client.h" +#include "xce/types.h" + +/* + * xceesd -- XCE External-resources Daemon (external-license-resources + * Phase 1, design.md 3.1). Joins the controller as HostRole_extres and + * pushes cluster-level FREE values for EXTERNAL shared resources into the + * existing resval pipeline (OPCode_update / Update_data_resvals, host=="" + * -- the sharedInst landing shape). + * + * Collectors are plugins over one internal interface; Phase 1 ships the + * script collector (any external resource: license-free wrappers, cloud + * quotas). The flexlm collector is the same interface over `lmstat -a` + * output parsing. + */ + +/* One configured collector: a name (must match a resources.yaml + * SHARED|GLOBAL|EXTERNAL resource) and how to get its value. */ +typedef struct esd_source_s { + char* name; /* resource name, same as resources.yaml */ + char* command; /* script collector: the command to run */ + int interval; /* poll interval, seconds (0: use the daemon default) */ + long stale_limit; /* seconds a kept value stays trustworthy (0: use + * the daemon default); design.md 3.4 */ + double last_value; /* last successfully-parsed value */ + time_t last_ok_time; /* when last_value was collected (0: never) */ + bool have_value; /* last_value is valid (any push has succeeded) */ + bool last_failed; /* last poll failed (stale-value policy input) */ +} esd_source_t; + +typedef struct esd_config_s { + char* controller_host; /* controller address (legacy single) */ + int controller_port; + char* peers; /* --peers style candidate list, or NULL */ + int poll_interval; /* daemon default poll interval, seconds */ + long stale_limit; /* daemon default stale limit, seconds */ + char* status_file; /* status JSON path (NULL: no file) */ + int nsrc; + esd_source_t* src; /* array of nsrc sources */ +} esd_config_t; + +/* Daemon-wide counters and identity for the status file (Phase 2, + * design.md 3.5 "xceadmin extres status"): the daemon fills one and hands + * it to esd_status_write each poll round. */ +typedef struct esd_stats_s { + time_t started_at; /* daemon start, epoch seconds */ + time_t now; /* snapshot time (filled by esd_status_write when 0) */ + long pushes_ok; /* successful resval pushes since start */ + long pushes_fail; /* failed pushes since start */ +} esd_stats_t; + +/* + * Parse esd.yaml: + * + * controller: 172.31.1.30:9990 # or peers: "h1:9990,h2:9990" + * poll_interval: 30 + * resources: + * - name: Verilog_XL + * collector: script + * command: /etc/xce/lic-report.sh + * interval: 15 # optional per-source override + * + * Returns 0 on success and fills *cfg_out (caller frees with + * esd_config_fini). Returns -errno-shaped negatives on failure: -EINVAL + * for a structurally-invalid file, -ENOENT for a missing one. Unknown + * keys are ignored (forward-compatible, same convention as the + * controller's own YAML loader). + */ +extern int esd_config_load(const char* path, esd_config_t* cfg_out); +extern void esd_config_fini(esd_config_t* cfg); + +/* + * Run one source's collector command and parse its output. The script + * collector contract (design.md 3.4): the command prints either + * " " (elim-style) or "" (a bare number), one line; + * leading/trailing whitespace is ignored. stdout only -- stderr is + * forwarded to the log untouched. + * + * Returns the parsed value, or NAN when the command failed or its output + * did not parse (caller decides the stale-value policy -- the daemon's + * default is keep-last-value + WARN, fail-open, design.md 3.4). + * + * posix_spawn(), not fork(): the daemon is multi-threaded (the client + * library runs its own transport thread), and a forked child that logged + * or allocated before exec could deadlock on a lock another thread held + * (tech.md "Process Creation"). The child gets an empty signal mask, + * default signal dispositions, and waits with a watchdog kill at + * esd_script_timeout_ms. + */ +extern double esd_script_collect(const char* command); + +/* Script-collector watchdog: how long a collection command may run before + * it is killed (default 10s; main.c overrides from the CLI). */ +extern int esd_script_timeout_ms; + +/* Default stale limit for sources (and the config default) -- design.md + * 3.4's example: 150s. A source whose collector keeps failing past this + * many seconds since its last good value reports stale=true in the status + * file (the daemon itself keeps the last value -- fail-open). */ +#define ESD_STALE_LIMIT_DEFAULT 150 + +/* Stale predicate (Phase 2): true when the source's last good value is + * older than its stale limit -- `now` vs src->last_ok_time, the source's + * own stale_limit when set, else the daemon default. A source with no + * value yet is never stale (there is nothing being kept alive; admission + * already treats the pool as empty through the normal NO_RESOURCE path). + * A last_ok_time in the future (clock skew) is not stale. */ +extern bool esd_source_is_stale(const esd_source_t* src, time_t now, + long default_limit); + +/* Write the daemon status JSON (Phase 2, "xceadmin extres status"): one + * object with the daemon identity, push counters, and per-source rows + * (name, last_value, last_ok_time, stale). `stats->now == 0` is filled + * with time(NULL). Written atomically (tmp + rename) so a reader never + * sees a partial file. Returns 0, or -errno-shaped negative (the caller + * logs; the status file is best-effort observability and must never take + * the poll loop down). */ +extern int esd_status_write(const char* path, const esd_config_t* cfg, + esd_stats_t* stats); + +#endif diff --git a/src/esd/meson.build b/src/esd/meson.build new file mode 100644 index 00000000..9bf4a993 --- /dev/null +++ b/src/esd/meson.build @@ -0,0 +1,42 @@ +# xceesd -- XCE External-resources Daemon (external-license-resources +# Phase 1, design.md 3.1). Joins the controller as HostRole_extres and +# pushes cluster-level FREE values into the existing resval pipeline. + +esd_srcs = [ + 'src/esd_config.c', + 'src/esd_script.c', + 'src/esd_status.c', + 'src/esd.c', +] + +esd_deps = [ + libxce_dep, + yaml_deps, + tinylog_dep, +] + +esd_incdir = include_directories('include', '../include', '../lib') + +xceesd = executable('xceesd', + esd_srcs + ['src/main.c'], + include_directories: [esd_incdir], + dependencies: esd_deps, + install: true) + +# Unit tests: white-box #include of the core sources (config parse, +# script-output parse, status writer, and -- since bug-1978 -- esd.c's +# reconnect-flag contract; esd_run's live-join sequence is still covered by +# the t1 E2E) -- same convention as the worker's resource_sync_test. +esd_test_incdir = include_directories('include', '../include', '../lib', '.') +esd_test_deps = [libxce_dep, yaml_deps, tinylog_dep, unity_dep] + +esd_test = executable('esd_test', + ['testsuite/test_esd_core.c'], + include_directories: [esd_test_incdir], + dependencies: esd_test_deps, + install: false) + +test('esd core test', esd_test, + suite: 'esd', + timeout: 60, + env: ['ASAN_OPTIONS=detect_leaks=0']) diff --git a/src/esd/src/esd.c b/src/esd/src/esd.c new file mode 100644 index 00000000..26996f70 --- /dev/null +++ b/src/esd/src/esd.c @@ -0,0 +1,363 @@ +/* esd.c -- the xceesd daemon body (external-license-resources Phase 1, + * design.md 3.1): join the controller as HostRole_extres, poll the + * configured collectors, and push every refreshed shared value as a + * host=="" resval delta -- the sharedInst landing shape the controller's + * opupd extres branch and the scheduler's DELTA sharedInst route expect. + * + * Reconnect semantics (design.md 3.5 step 6, "重连即全量重推"): every + * (re)join is followed by a FULL push of every source's last-known value + * -- the same contract the worker's job-state resync re-report has + * (9d7628e): the controller keeps no durable state for this daemon, so a + * fresh controller (boot or failover promotion) must be told everything + * before the next poll tick. + */ +#include +#include +#include +#include + +#include "esd.h" +#include "tlog.h" +#include "xce/cluster.h" + +/* The daemon's client-library context and identity (from main.c's CLI). */ +typedef struct esd_daemon_s { + xcecli_ctx_t apictx; + esd_config_t cfg; + const char* from; /* join identity ("xceesd") */ + int running; /* 0 once the main loop should exit */ + xce_cluster_config_t cluster_cfg; /* validated peers list (HA mode) */ + esd_stats_t stats; /* status-file counters (Phase 2) */ +} esd_daemon_t; + +static esd_daemon_t g_esd; + +int esd_push_value(const char* name, double value) +{ + xce_resval_t resval; + xce_resval_t* valptr = &resval; /* the wire takes an array of POINTERS + * (xce_resval_t**, typemap.h) -- passing + * &resval itself made the encoder read + * the struct's first word as the next + * element pointer and strlen() garbage + * (live segfault, 2026-10-26) */ + xce_val_t val; + int rc; + + memset(&resval, 0, sizeof(resval)); + memset(&val, 0, sizeof(val)); + + val.which = Value_u_fvalue; + val.u.fvalue = value; + + /* host == "" is THE landing-shape marker: the controller's opupd + * extres branch drops the worker lookup entirely and routes the entry + * to update_val_res's SHARED branch (c->sharedinst), and the + * scheduler's DELTA path treats host==""/NULL as sharedInst (mirrors + * snapshot_load.c's bug-1923 precedent). */ + resval.name = (char*)name; + resval.host = (char*)""; + resval.val = &val; + + rc = xcecli_update_resvals(g_esd.apictx, 1, &valptr); + if (rc < 0) { + ERROR("esd push %s=%f failed (rc=%d)", name, value, rc); + g_esd.stats.pushes_fail++; + } else { + TRACE("esd pushed %s=%f", name, value); + g_esd.stats.pushes_ok++; + } + + return rc; +} + +/* One poll pass: collect every source, remember the last good value, and + * push only what CHANGED (the debounce -- design.md step 6 "轮询去抖"). + * A failed collection keeps the previous value and does not push (fail- + * open: the controller keeps scheduling against the last known good + * count; design.md 3.4 stale policy default). + * + * Phase 2: every source records last_ok_time on a good collect (the + * stale predicate's input, exposed via the status file); a failure past + * the source's stale limit escalates from the one-shot WARN to a per- + * round stale WARN -- the operator sees a kept-old-value daemon without + * having to guess from the absence of "pushed" lines. */ +int esd_poll_round(void) +{ + int i; + int pushed = 0; + time_t now = time(NULL); + + for (i = 0; i < g_esd.cfg.nsrc; i++) { + esd_source_t* src = &(g_esd.cfg.src[i]); + double v; + + v = esd_script_collect(src->command); + + if (isnan(v)) { + if (!src->last_failed) { + WARN("source %s: collection failed -- keeping last known " + "value %f (fail-open)", + src->name, src->have_value ? src->last_value : 0.0); + } else if (esd_source_is_stale(src, now, + g_esd.cfg.stale_limit)) { + WARN("source %s: STALE for %lds (limit %lds) -- last good " + "value %f kept (fail-open); check the collector", + src->name, + (long)(now - src->last_ok_time), + (src->stale_limit > 0) ? src->stale_limit + : g_esd.cfg.stale_limit, + src->last_value); + } + src->last_failed = true; + continue; + } + + src->last_failed = false; + src->last_ok_time = now; + + if (src->have_value && (src->last_value == v)) { + continue; /* unchanged -- the debounce */ + } + + if (esd_push_value(src->name, v) == 0) { + src->have_value = true; + src->last_value = v; + pushed++; + } + } + + return pushed; +} + +/* Full re-push of every source's last-known value (reconnect semantics: + * the controller we just joined knows nothing about us). Sources with no + * value yet are skipped -- nothing to push, the poll round will fill + * them. */ +int esd_push_all(void) +{ + int i; + int pushed = 0; + + for (i = 0; i < g_esd.cfg.nsrc; i++) { + esd_source_t* src = &(g_esd.cfg.src[i]); + + if (!src->have_value) { + continue; + } + + if (esd_push_value(src->name, src->last_value) == 0) { + pushed++; + } + } + + INFO("esd full re-push: %d/%d source(s)", pushed, g_esd.cfg.nsrc); + + return pushed; +} + +/* --rejoin hook (client library contract): the library calls this while + * holding its failover lock -- no I/O, no xcecli_* calls (would self- + * deadlock, the worker's xcecd_on_reconnect lesson, controller_connector.c). + * We only set flags; the main loop consumes them after the lock is free. */ +static volatile int g_needs_full_push = 0; +static volatile int g_needs_rejoin = 0; + +static int esd_on_reconnect(xcecli_ctx_t ctx, void* arg) +{ + (void)ctx; + (void)arg; + + /* bug-1978: a re-PUSH alone is not enough -- the fresh controller + * endpoint was created by handle_connection() with priv->role = + * HostRole_client, and only OPCode_join sets it to HostRole_extres + * (join.c). Without the re-join every resval push lands in + * do_update_resvals's neither-branch and is refused -EPERM forever + * (the live sighting: pushes_ok frozen, rc=-1 on every attempt after + * a controller restart). Mirror the worker's needs_rejoin contract: + * the main loop re-issues the role join, then does the full re-push. */ + g_needs_rejoin = 1; + g_needs_full_push = 1; + + return 0; +} + +int esd_run(void) +{ + int next_full_push = 1; /* the FIRST join needs the full push too */ + + /* Join sequence mirrors the worker's xcecd_join_cluster (controller_ + * connector.c): build ONE validated candidate list -- peers from + * esd.yaml when given, else the single controller host:port -- and + * hand it to the client library, which owns failover rotation and + * standby redirects inside the single xcecli_join call. */ + { + xce_client_config_t cconfig; + + memset(&cconfig, 0, sizeof(cconfig)); + + if (g_esd.cfg.peers != NULL) { + int rc = xce_parse_peers(g_esd.cfg.peers, &g_esd.cluster_cfg); + if (rc < 0) { + ERROR("esd: cannot parse peers \"%s\"", g_esd.cfg.peers); + return -EINVAL; + } + strncpy(g_esd.cluster_cfg.identity, g_esd.from, + sizeof(g_esd.cluster_cfg.identity) - 1); + rc = xce_cluster_config_validate(&g_esd.cluster_cfg); + if (rc < 0) { + ERROR("esd: invalid peers config (rc=%d)", rc); + return rc; + } + rc = xce_cluster_config_to_client_config(&g_esd.cluster_cfg, + &cconfig); + if (rc < 0) { + ERROR("esd: candidate list conversion failed (rc=%d)", rc); + return rc; + } + } else { + cconfig.nmasters = 1; + cconfig.masters[0] = g_esd.cfg.controller_host; + cconfig.master_ports[0] = g_esd.cfg.controller_port; + } + + cconfig.from = (char*)g_esd.from; + cconfig.key = (char*)"keepsecret$"; + + { + int rc = xcecli_ctx_init(&g_esd.apictx, &cconfig); + if (rc < 0) { + ERROR("esd: client ctx init failed (rc=%d)", rc); + return rc; + } + } + + if (xcecli_start(g_esd.apictx) < 0) { + ERROR("esd: cannot reach controller %s:%d", + g_esd.cfg.controller_host, g_esd.cfg.controller_port); + xcecli_ctx_fini(g_esd.apictx); + return -ECONNREFUSED; + } + + /* AFTER xcecli_start(): start() resets rejoin_cb (the worker's + * task-2.3 lesson, controller_connector.c). */ + xcecli_set_reconnect_handler(g_esd.apictx, esd_on_reconnect, NULL); + + { + int rc = xcecli_join(g_esd.apictx, HostRole_extres, NULL); + if (rc < 0) { + ERROR("esd: join failed (rc=%d)", rc); + xcecli_stop(g_esd.apictx); + xcecli_ctx_fini(g_esd.apictx); + return rc; + } + } + } + + INFO("esd: joined cluster as %s (%d source(s))", g_esd.from, + g_esd.cfg.nsrc); + + while (g_esd.running) { + int i; + + if (g_needs_rejoin) { + /* bug-1978: a transparent reconnect just landed -- the fresh + * controller endpoint defaults to HostRole_client, so the role + * join must be re-issued BEFORE any push (an un-joined push is + * refused -EPERM). This runs on the main loop, outside the + * library's failover lock (the xcecd_on_reconnect constraint). + * Failure keeps the flag set: the next tick retries, mirroring + * the worker's needs_rejoin self-healing cadence. rc >= 0 is + * success for the same reason as the worker's rejoin (the + * controller's reply.status for a plain extres join is 0, but + * the failover wrapper surfaces its own codes on success). */ + int rc = xcecli_join(g_esd.apictx, HostRole_extres, NULL); + if (rc >= 0) { + g_needs_rejoin = 0; + INFO("esd: re-joined after reconnect (rc=%d)", rc); + } else { + WARN("esd: re-join after reconnect failed (rc=%d) -- " + "retrying next tick", + rc); + } + } + + if (g_needs_full_push) { + /* A transparent reconnect just landed (new primary after + * failover, or the old controller restarted): it has no + * record of this daemon -- push everything we know. */ + g_needs_full_push = 0; + next_full_push = 0; + (void)esd_push_all(); + } + + if (next_full_push) { + next_full_push = 0; + (void)esd_push_all(); + } + + /* bug (live sighting 2026-10-26): with every source unchanged, the + * debounce suppresses ALL pushes -- and a reconnect only ever fires + * INSIDE a push's failover call. A controller restart while our + * value sits still left the daemon idle on a dead connection + * forever (no socket() for 50+ minutes, pushes never resumed). + * The broken-connection flag is set asynchronously by the + * transport's RDHUP handler (bug-1760's pure predicate): when it + * latches, force one full re-push -- the write attempt triggers the + * rotate/reconnect, and esd_on_reconnect's flags keep the + * re-join + full re-push contract for the failover path too + * (bug-1978: the re-push alone was refused -EPERM on the un-joined + * fresh endpoint). */ + if (g_esd.cfg.nsrc > 0 && xcecli_is_connection_broken(g_esd.apictx)) { + INFO("esd: connection broken -- forcing a full re-push to " + "reconnect"); + (void)esd_push_all(); + } + + (void)esd_poll_round(); + + /* Phase 2: refresh the status file every round (best-effort; the + * poll loop must survive a bad path). */ + if (g_esd.cfg.status_file != NULL) { + /* one WARN per failure would spam a broken path every round; + * log on the transition only */ + static bool status_warned = false; + int src = esd_status_write(g_esd.cfg.status_file, &g_esd.cfg, + &g_esd.stats); + if (src < 0) { + if (!status_warned) { + WARN("esd: cannot write status file %s (rc=%d) -- " + "continuing without it", + g_esd.cfg.status_file, src); + status_warned = true; + } + } else { + status_warned = false; + } + } + + /* sleep in 1s slices so a signal stop lands promptly */ + for (i = 0; (i < g_esd.cfg.poll_interval) && g_esd.running; i++) { + sleep(1); + } + } + + xcecli_stop(g_esd.apictx); + xcecli_ctx_fini(g_esd.apictx); + + return 0; +} + +void esd_stop(void) +{ + g_esd.running = 0; +} + +void esd_init(esd_config_t* cfg, const char* from) +{ + memset(&g_esd, 0, sizeof(g_esd)); + g_esd.cfg = *cfg; + g_esd.from = from; + g_esd.running = 1; + g_esd.stats.started_at = time(NULL); +} diff --git a/src/esd/src/esd_config.c b/src/esd/src/esd_config.c new file mode 100644 index 00000000..5ad145c7 --- /dev/null +++ b/src/esd/src/esd_config.c @@ -0,0 +1,311 @@ +/* esd_config.c -- esd.yaml parsing for xceesd (external-license-resources + * Phase 1, design.md 3.4). libcyaml, same loader pattern as the + * autoscaler's prov_conf.c: schema-driven, unknown keys ignored + * (CYAML_CFG_IGNORE_UNKNOWN_KEYS), explicit field validation after load + * (a source without name or command can never report -- fail loudly at + * load, not silently at runtime). */ +#include +#include +#include +#include + +#include + +#include "esd.h" + +typedef struct yaml_esd_source_s { + char* name; + char* collector; + char* command; + long interval; /* 0 = daemon default */ + long stale_limit; /* 0 = daemon default (Phase 2, design.md 3.4) */ +} yaml_esd_source_t; + +typedef struct yaml_esd_config_s { + char* controller; + char* peers; + long poll_interval; + long stale_limit; /* 0 = ESD_STALE_LIMIT_DEFAULT */ + char* status_file; /* NULL = no status file */ + yaml_esd_source_t* resources; + unsigned resources_count; +} yaml_esd_config_t; + +static const cyaml_schema_field_t esd_source_fields_schema[] = { + CYAML_FIELD_STRING_PTR("name", CYAML_FLAG_POINTER, yaml_esd_source_t, + name, 1, CYAML_UNLIMITED), + CYAML_FIELD_STRING_PTR("collector", CYAML_FLAG_POINTER, yaml_esd_source_t, + collector, 1, CYAML_UNLIMITED), + CYAML_FIELD_STRING_PTR("command", CYAML_FLAG_POINTER, yaml_esd_source_t, + command, 1, CYAML_UNLIMITED), + CYAML_FIELD_INT("interval", CYAML_FLAG_OPTIONAL, yaml_esd_source_t, + interval), + CYAML_FIELD_INT("stale_limit", CYAML_FLAG_OPTIONAL, yaml_esd_source_t, + stale_limit), + CYAML_FIELD_END +}; + +static const cyaml_schema_value_t esd_source_schema = { + CYAML_VALUE_MAPPING(CYAML_FLAG_DEFAULT, yaml_esd_source_t, + esd_source_fields_schema) +}; + +static const cyaml_schema_field_t esd_config_fields_schema[] = { + CYAML_FIELD_STRING_PTR("controller", CYAML_FLAG_OPTIONAL, + yaml_esd_config_t, controller, 1, CYAML_UNLIMITED), + CYAML_FIELD_STRING_PTR("peers", CYAML_FLAG_OPTIONAL, yaml_esd_config_t, + peers, 1, CYAML_UNLIMITED), + CYAML_FIELD_INT("poll_interval", CYAML_FLAG_OPTIONAL, yaml_esd_config_t, + poll_interval), + CYAML_FIELD_INT("stale_limit", CYAML_FLAG_OPTIONAL, yaml_esd_config_t, + stale_limit), + CYAML_FIELD_STRING_PTR("status_file", CYAML_FLAG_OPTIONAL, + yaml_esd_config_t, status_file, 1, CYAML_UNLIMITED), + CYAML_FIELD_SEQUENCE("resources", CYAML_FLAG_POINTER, yaml_esd_config_t, + resources, &esd_source_schema, 0, CYAML_UNLIMITED), + CYAML_FIELD_END +}; + +static const cyaml_schema_value_t esd_config_schema = { + CYAML_VALUE_MAPPING(CYAML_FLAG_POINTER, yaml_esd_config_t, + esd_config_fields_schema) +}; + +/* "host:port" -> (host, port); returns 0 on success, -EINVAL when port + * missing/non-numeric/out of range. Mutates `spec` (splits at the colon). */ +static int esd_parse_endpoint(char* spec, char** host_out, int* port_out) +{ + char* colon; + char* endp = NULL; + long port; + + if ((spec == NULL) || (*spec == '\0')) { + return -EINVAL; + } + + colon = strrchr(spec, ':'); + if ((colon == NULL) || (colon[1] == '\0')) { + return -EINVAL; + } + + port = strtol(colon + 1, &endp, 10); + if ((endp == NULL) || (*endp != '\0') || (port <= 0) || (port > 65535)) { + return -EINVAL; + } + + *colon = '\0'; + *host_out = spec; + *port_out = (int)port; + + return 0; +} + +/* File-content variant, split out from the file loader for testability: + * the tests run esd_config_load_path() against real files, so both layers + * stay production code. */ +int esd_config_load_path(const char* dir, const char* name, + esd_config_t* cfg_out) +{ + static cyaml_config_t cyaml_cfg = { + .mem_fn = cyaml_mem, + .log_level = CYAML_LOG_WARNING, + .flags = CYAML_CFG_IGNORE_UNKNOWN_KEYS, + }; + + yaml_esd_config_t* ycfg = NULL; + cyaml_err_t err; + char* path = NULL; + esd_config_t cfg; + int i; + int rc; + + if ((dir == NULL) || (name == NULL) || (cfg_out == NULL)) { + return -EINVAL; + } + + memset(cfg_out, 0, sizeof(*cfg_out)); + memset(&cfg, 0, sizeof(cfg)); + + if (asprintf(&path, "%s/%s", dir, name) < 0) { + return -ENOMEM; + } + + err = cyaml_load_file(path, &cyaml_cfg, &esd_config_schema, + (void**)&ycfg, NULL); + free(path); + if (err != CYAML_OK) { + /* libcyaml collapses file-not-found into generic errors; map the + * obvious one so the daemon's startup log can say "missing file" + * vs "bad file" honestly. */ + char probe[512]; + FILE* f; + + snprintf(probe, sizeof(probe), "%s/%s", dir, name); + f = fopen(probe, "r"); + if (f == NULL) { + return -ENOENT; + } + fclose(f); + return -EINVAL; + } + + if (ycfg == NULL) { + return -EINVAL; + } + + /* Validation first: no controller target and no peers list means the + * daemon could never connect to anything. */ + if ((ycfg->controller == NULL) && (ycfg->peers == NULL)) { + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + return -EINVAL; + } + + if (ycfg->controller != NULL) { + char* host = NULL; + int port = 0; + + if (esd_parse_endpoint(ycfg->controller, &host, &port) < 0) { + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + return -EINVAL; + } + + /* esd_parse_endpoint splits ycfg->controller in place -- the host + * pointer points INTO the cyaml-owned string. Copy out before the + * cyaml_free below releases the whole decoded tree. */ + cfg.controller_host = strdup(host); + cfg.controller_port = port; + if (cfg.controller_host == NULL) { + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + return -ENOMEM; + } + } + + cfg.peers = (ycfg->peers != NULL) ? strdup(ycfg->peers) : NULL; + cfg.poll_interval = (int)ycfg->poll_interval; + cfg.stale_limit = ycfg->stale_limit; + cfg.status_file = (ycfg->status_file != NULL) ? strdup(ycfg->status_file) + : NULL; + cfg.nsrc = (int)ycfg->resources_count; + cfg.src = NULL; + + if ((cfg.peers == NULL) && (ycfg->peers != NULL)) { + esd_config_fini(&cfg); + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + return -ENOMEM; + } + if ((cfg.status_file == NULL) && (ycfg->status_file != NULL)) { + esd_config_fini(&cfg); + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + return -ENOMEM; + } + + if (cfg.nsrc > 0) { + cfg.src = (esd_source_t*)calloc((size_t)cfg.nsrc, + sizeof(esd_source_t)); + if (cfg.src == NULL) { + esd_config_fini(&cfg); + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + return -ENOMEM; + } + } + + rc = 0; + for (i = 0; i < cfg.nsrc; i++) { + yaml_esd_source_t* ys = &(ycfg->resources[i]); + + if ((ys->name == NULL) || (ys->command == NULL)) { + rc = -EINVAL; + break; + } + + cfg.src[i].name = strdup(ys->name); + cfg.src[i].command = strdup(ys->command); + cfg.src[i].interval = (int)ys->interval; + cfg.src[i].stale_limit = ys->stale_limit; + + if ((cfg.src[i].name == NULL) || (cfg.src[i].command == NULL)) { + rc = -ENOMEM; + break; + } + } + + cyaml_free(&cyaml_cfg, &esd_config_schema, ycfg, 0); + + if (rc < 0) { + esd_config_fini(&cfg); + return rc; + } + + *cfg_out = cfg; + + return 0; +} + +int esd_config_load(const char* path, esd_config_t* cfg_out) +{ + char* dir; + char* name; + char* slash; + int rc; + + if ((path == NULL) || (cfg_out == NULL)) { + return -EINVAL; + } + + dir = strdup(path); + if (dir == NULL) { + return -ENOMEM; + } + + slash = strrchr(dir, '/'); + if (slash == NULL) { + name = dir; + dir = strdup("."); + if (dir == NULL) { + free(name); + return -ENOMEM; + } + } else { + *slash = '\0'; + name = slash + 1; + if (*name == '\0') { + free(dir); + return -EINVAL; + } + if (dir == name - 1) { + /* path was "/name" -- dir is "" */ + free(dir); + dir = strdup("/"); + if (dir == NULL) { + return -ENOMEM; + } + } + } + + rc = esd_config_load_path(dir, name, cfg_out); + + free(dir); + + return rc; +} + +void esd_config_fini(esd_config_t* cfg) +{ + int i; + + if (cfg == NULL) { + return; + } + + free(cfg->controller_host); + free(cfg->peers); + free(cfg->status_file); + + for (i = 0; i < cfg->nsrc; i++) { + free(cfg->src[i].name); + free(cfg->src[i].command); + } + free(cfg->src); + + memset(cfg, 0, sizeof(*cfg)); +} diff --git a/src/esd/src/esd_script.c b/src/esd/src/esd_script.c new file mode 100644 index 00000000..fc1827b4 --- /dev/null +++ b/src/esd/src/esd_script.c @@ -0,0 +1,230 @@ +/* esd_script.c -- the script collector for xceesd (external-license- + * resources Phase 1, design.md 3.1): runs a user command with posix_spawn, + * reads its stdout, and parses "" or " " from it. + * + * posix_spawn(), not fork(): the daemon is multi-threaded (the client + * library runs its own transport thread), and a forked child that logged + * or allocated before exec could deadlock on a lock another thread held + * (tech.md "Process Creation" -- same rationale as the autoscaler's + * plug_mgr.c run_script). The child starts with an empty signal mask and + * default signal dispositions. + * + * The watchdog: the pipe read is bounded by esd_script_timeout_ms -- + * a hung collection command must not stall the poll loop (the value it + * was going to report belongs to a dead world anyway). We enforce it + * with poll(2) on the read end + a kill() at deadline; waitpid reaps. + */ +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "esd.h" +#include "tlog.h" + +/* Script-collector watchdog, seconds of wall clock a collection command + * may run before it is killed (design.md 3.4: the poll loop must survive + * a hung script). Overridable from main.c. */ +int esd_script_timeout_ms = 10 * 1000; + +/* + * esd_parse_script_output -- the collector contract (design.md 3.4): the + * first non-empty line carries either a bare number or " ". + * Leading/trailing whitespace ignored; anything else is garbage -> NAN + * (the caller owns the stale-value policy). + */ +double esd_parse_script_output(const char* out) +{ + const char* p; + char* endp = NULL; + double v; + + if (out == NULL) { + return NAN; + } + + p = out; + while (*p != '\0') { + /* skip horizontal whitespace and newlines */ + while ((*p == ' ') || (*p == '\t') || (*p == '\r') || (*p == '\n')) { + p++; + } + if (*p == '\0') { + break; + } + + v = strtod(p, &endp); + if ((endp != NULL) && (endp != p)) { + /* accept trailing junk after the number (a name may follow in + * "value name" order too -- the number is what we need) */ + return v; + } + + /* skip this non-numeric token and try the next one */ + while ((*p != '\0') && (*p != ' ') && (*p != '\t') && (*p != '\r') + && (*p != '\n')) { + p++; + } + } + + return NAN; +} + +/* The argv for the child: run the command via sh -c, so a plain + * "/path/script.sh arg1 arg2" line works without the daemon ever + * splitting shell words itself (the operator owns quoting). */ +static int esd_script_spawn(const char* command, int out_fd, pid_t* pid_out) +{ + posix_spawn_file_actions_t fa; + posix_spawnattr_t attr; + sigset_t none, all; + char* argv[] = { (char*)"sh", (char*)"-c", (char*)command, NULL }; + extern char** environ; + pid_t pid; + int rc; + + sigemptyset(&none); + sigfillset(&all); + + rc = posix_spawn_file_actions_init(&fa); + if (rc == 0) { + rc = posix_spawn_file_actions_adddup2(&fa, out_fd, STDOUT_FILENO); + } + if (rc == 0) { + rc = posix_spawn_file_actions_addopen(&fa, STDERR_FILENO, "/dev/null", + O_WRONLY, 0); + } + if (rc == 0) { + rc = posix_spawnattr_init(&attr); + } + if (rc == 0) { + rc = posix_spawnattr_setflags( + &attr, POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF); + } + if (rc == 0) { + rc = posix_spawnattr_setsigmask(&attr, &none); + } + if (rc == 0) { + rc = posix_spawnattr_setsigdefault(&attr, &all); + } + if (rc == 0) { + rc = posix_spawn(&pid, "/bin/sh", &fa, &attr, argv, environ); + } + posix_spawnattr_destroy(&attr); + posix_spawn_file_actions_destroy(&fa); + + if (rc != 0) { + return -rc; + } + + *pid_out = pid; + + return 0; +} + +double esd_script_collect(const char* command) +{ + int fds[2]; + pid_t pid = -1; + double value = NAN; + char buf[4096]; + size_t total = 0; + int elapsed_ms = 0; + int rc; + int wstatus = 0; + struct pollfd pfd; + + if ((command == NULL) || (*command == '\0')) { + return NAN; + } + + if (pipe(fds) != 0) { + ERROR("script collect: pipe failed (%s)", strerror(errno)); + return NAN; + } + + rc = esd_script_spawn(command, fds[1], &pid); + close(fds[1]); /* parent's copy -- child holds the write end now */ + if (rc < 0) { + close(fds[0]); + ERROR("script collect: spawn failed for \"%s\" (%s)", command, + strerror(-rc)); + return NAN; + } + + /* Bound the read at the watchdog deadline: a script that hangs (a + * blocked lmstat, a wedged NFS mount) must not stall the poll loop. */ + pfd.fd = fds[0]; + pfd.events = POLLIN; + + while (total < sizeof(buf) - 1) { + int pret = poll(&pfd, 1, 100); + + if (pret < 0) { + if (errno == EINTR) { + continue; + } + ERROR("script collect: poll failed (%s)", strerror(errno)); + break; + } + if (pret == 0) { + elapsed_ms += 100; + if (elapsed_ms >= esd_script_timeout_ms) { + WARN("script collect: command \"%s\" exceeded %d ms -- " + "killing it, value not refreshed this round", + command, esd_script_timeout_ms); + kill(pid, SIGKILL); + waitpid(pid, &wstatus, 0); + close(fds[0]); + return NAN; + } + continue; + } + + if (pfd.revents & (POLLERR | POLLHUP)) { + ssize_t n = read(fds[0], buf + total, sizeof(buf) - 1 - total); + if (n > 0) { + total += (size_t)n; + } + break; + } + if (pfd.revents & POLLIN) { + ssize_t n = read(fds[0], buf + total, sizeof(buf) - 1 - total); + if (n <= 0) { + break; /* EOF or error: done reading */ + } + total += (size_t)n; + } + } + + buf[total] = '\0'; + close(fds[0]); + + /* reap regardless of which path brought us here */ + while (waitpid(pid, &wstatus, 0) < 0) { + if (errno != EINTR) { + break; + } + } + + if (total == 0) { + WARN("script collect: command \"%s\" produced no output", command); + return NAN; + } + + value = esd_parse_script_output(buf); + if (isnan(value)) { + WARN("script collect: command \"%s\" output did not parse as a " + "value (got %.60s)", command, buf); + } + + return value; +} diff --git a/src/esd/src/esd_status.c b/src/esd/src/esd_status.c new file mode 100644 index 00000000..9a39fbba --- /dev/null +++ b/src/esd/src/esd_status.c @@ -0,0 +1,123 @@ +/* esd_status.c -- xceesd status file (external-license-resources Phase 2, + * design.md 3.5 "xceadmin extres status"). The daemon writes one JSON + * snapshot per poll round to a configured path; xceadmin reads it on the + * daemon host. No new wire opcode: the PR-2 proxy formalizes a status + * channel later, and a local file keeps this observability surface + * strictly read-only and crash-safe. + * + * Atomicity: write to .tmp then rename(2) -- a concurrent reader + * (xceadmin) either sees the previous complete file or the new one, never + * a partial write (the fake-license E2E's lic_state lesson: a reader + * observing a truncated file mistakes it for data loss). + */ +#include +#include +#include +#include +#include +#include + +#include "esd.h" + +bool esd_source_is_stale(const esd_source_t* src, time_t now, + long default_limit) +{ + long limit; + long age; + + if ((src == NULL) || (!src->have_value) || (src->last_ok_time == 0)) { + return false; + } + + /* Future last_ok_time (caller clock skew): age would go negative and + * read as fresh -- that is the honest reading, keep it by clamping. */ + age = (now > src->last_ok_time) ? (long)(now - src->last_ok_time) : 0; + + limit = (src->stale_limit > 0) ? src->stale_limit : default_limit; + if (limit <= 0) { + limit = ESD_STALE_LIMIT_DEFAULT; + } + + return age > limit; +} + +int esd_status_write(const char* path, const esd_config_t* cfg, + esd_stats_t* stats) +{ + char tmp[1024]; + FILE* f; + int i; + time_t now; + int rc = 0; + + if ((path == NULL) || (cfg == NULL) || (stats == NULL)) { + return -EINVAL; + } + + now = (stats->now != 0) ? stats->now : time(NULL); + + snprintf(tmp, sizeof(tmp), "%s.tmp", path); + f = fopen(tmp, "w"); + if (f == NULL) { + return -errno; + } + + if (fprintf(f, + "{\n" + " \"daemon\": \"xceesd\",\n" + " \"pid\": %ld,\n" + " \"started_at\": %ld,\n" + " \"generated_at\": %ld,\n" + " \"pushes_ok\": %ld,\n" + " \"pushes_fail\": %ld,\n" + " \"sources\": [\n", + (long)getpid(), (long)stats->started_at, (long)now, + stats->pushes_ok, stats->pushes_fail) < 0) { + rc = -EIO; + goto out; + } + + for (i = 0; i < cfg->nsrc; i++) { + const esd_source_t* src = &(cfg->src[i]); + bool stale = + esd_source_is_stale(src, now, cfg->stale_limit); + char valbuf[64]; + + if (src->have_value) { + snprintf(valbuf, sizeof(valbuf), "%.17g", src->last_value); + } else { + snprintf(valbuf, sizeof(valbuf), "null"); + } + + if (fprintf(f, + " {\"name\": \"%s\", \"last_value\": %s, " + "\"last_ok_time\": %ld, \"stale\": %s}%s\n", + (src->name != NULL) ? src->name : "", + valbuf, + (long)src->last_ok_time, stale ? "true" : "false", + (i + 1 < cfg->nsrc) ? "," : "") < 0) { + rc = -EIO; + goto out; + } + } + + if (fprintf(f, " ]\n}\n") < 0) { + rc = -EIO; + goto out; + } + +out: + if (fclose(f) != 0) { + rc = (rc == 0) ? -EIO : rc; + } + if (rc == 0) { + if (rename(tmp, path) != 0) { + rc = -errno; + } + } + if (rc != 0) { + unlink(tmp); /* never leave the .tmp behind for a reader to trip on */ + } + + return rc; +} diff --git a/src/esd/src/main.c b/src/esd/src/main.c new file mode 100644 index 00000000..1cd1811d --- /dev/null +++ b/src/esd/src/main.c @@ -0,0 +1,127 @@ +/* main.c -- xceesd entry point (external-license-resources Phase 1, + * design.md 3.1). Minimal on purpose: parse the CLI, load esd.yaml, run + * the poll loop until SIGINT/SIGTERM. */ +#include +#include +#include +#include +#include +#include + +#include "esd.h" +#include "tlog.h" + +static char doc[] = "xceesd -- XCE External-resources Daemon"; +static char args_doc[] = ""; + +const char* argp_program_bug_address = BUG_EMAIL; + +struct arguments { + char* logfile; + char* config; + int verbose; + int timeout_ms; +}; + +#define OPT_TIMEOUT 256 + +static struct argp_option options[] = { + { "logfile", 'l', "FILE", 0, "The full path of log file" }, + { "config", 'c', "FILE", 0, + "Path of the daemon configuration file (default: /etc/xce/esd.yaml)" }, + { "verbose", 'v', 0, 0, "Produce verbose log" }, + { "script-timeout", OPT_TIMEOUT, "MS", 0, + "Watchdog for collection commands in milliseconds (default: 10000)" }, + { 0 } +}; + +static error_t parse_opt(int key, char* arg, struct argp_state* state) +{ + struct arguments* state_args = state->input; + + switch (key) { + case 'l': + state_args->logfile = arg; + break; + case 'c': + state_args->config = arg; + break; + case 'v': + state_args->verbose++; + break; + case OPT_TIMEOUT: + state_args->timeout_ms = atoi(arg); + break; + default: + return ARGP_ERR_UNKNOWN; + } + + return 0; +} + +static struct argp argp = { options, parse_opt, args_doc, doc }; + +static esd_config_t g_cfg; + +static void sig_handler(int signo) +{ + (void)signo; + esd_stop(); +} + +int main(int argc, char* argv[]) +{ + struct arguments arguments; + char* logfile = "/tmp/xceesd.log"; + char* loglevel = "info"; + char* config = "/etc/xce/esd.yaml"; + int rc; + + memset(&arguments, 0, sizeof(arguments)); + argp_parse(&argp, argc, argv, 0, 0, &arguments); + + if (arguments.logfile != NULL) { + logfile = arguments.logfile; + } + if (arguments.config != NULL) { + config = arguments.config; + } + if (arguments.verbose > 0) { + loglevel = "debug"; + } + if (arguments.timeout_ms > 0) { + esd_script_timeout_ms = arguments.timeout_ms; + } + + mylog_init(logfile, loglevel, 16 * 1024 * 1024, 4); + + rc = esd_config_load(config, &g_cfg); + if (rc < 0) { + fprintf(stderr, "xceesd: cannot load config %s: %s\n", config, + (rc == -ENOENT) ? "file not found" + : (rc == -EINVAL) ? "invalid file" + : strerror(-rc)); + return 1; + } + + INFO("xceesd starting: config %s, %d source(s), poll %ds", config, + g_cfg.nsrc, g_cfg.poll_interval > 0 ? g_cfg.poll_interval : 30); + + if (g_cfg.poll_interval <= 0) { + g_cfg.poll_interval = 30; + } + + signal(SIGINT, sig_handler); + signal(SIGTERM, sig_handler); + + esd_init(&g_cfg, "xceesd"); + + rc = esd_run(); + + esd_config_fini(&g_cfg); + mylog_fini(); + + INFO("xceesd stopped (rc=%d)", rc); + + return (rc == 0) ? 0 : 1; +} diff --git a/src/esd/testsuite/test_esd_core.c b/src/esd/testsuite/test_esd_core.c new file mode 100644 index 00000000..2ad03a85 --- /dev/null +++ b/src/esd/testsuite/test_esd_core.c @@ -0,0 +1,434 @@ +/* test_esd_core.c -- external-license-resources Phase 1, step 6 TDD: the + * xceesd skeleton's pure core -- esd.yaml parsing and the script + * collector's output parsing -- white-box #include of the sources under + * test (same pattern as the worker's resource_sync_test / + * controller_connector_test). The join/reconnect wiring lives in esd.c + * against the live client library and is exercised by the t1 E2E, not + * here; everything this file drives must be deterministic. */ +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "esd.h" + +/* White-box: reach the config schema/parse internals. */ +#include "../src/esd_config.c" +/* White-box: the script collector's parse half (esd_parse_script_output); + * the spawn half (esd_script_collect) is exercised by the t1 E2E with a + * real fake-license script, not here. */ +#include "../src/esd_script.c" +/* White-box: the stale predicate and the status-file writer (Phase 2, + * design.md 3.5 "运维面"). */ +#include "../src/esd_status.c" +/* White-box: the reconnect-flag contract (bug-1978). esd.c's main loop + * drives the live client library, but the reconnect handler's flag contract + * -- set BOTH rejoin and full-push, and the loop consuming rejoin first -- + * is unit-testable here without a controller: esd_run() itself is not + * linked (its join sequence would need a live primary); we drive + * esd_on_reconnect and read the loop's guard statics directly. */ +#include "../src/esd.c" + +static char g_tmpdir[256]; + +void setUp(void) +{ + snprintf(g_tmpdir, sizeof(g_tmpdir), "/tmp/xce_esd_testXXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(g_tmpdir)); +} + +void tearDown(void) +{ + if (g_tmpdir[0] != '\0') { + char path[300]; + + /* tests only write flat files inside g_tmpdir */ + snprintf(path, sizeof(path), "rm -rf %s", g_tmpdir); + if (system(path) != 0) { + /* best effort */ + } + g_tmpdir[0] = '\0'; + } +} + +static void write_file(const char* dir, const char* name, const char* content) +{ + char path[512]; + FILE* f; + + snprintf(path, sizeof(path), "%s/%s", dir, name); + f = fopen(path, "w"); + TEST_ASSERT_NOT_NULL(f); + fputs(content, f); + fclose(f); +} + +static char* slurp(const char* path) +{ + FILE* f; + long len; + char* buf; + + f = fopen(path, "r"); + if (f == NULL) { + return NULL; + } + fseek(f, 0, SEEK_END); + len = ftell(f); + fseek(f, 0, SEEK_SET); + buf = (char*)malloc((size_t)len + 1); + if (buf == NULL) { + fclose(f); + return NULL; + } + if (fread(buf, 1, (size_t)len, f) != (size_t)len) { + fclose(f); + free(buf); + return NULL; + } + buf[len] = '\0'; + fclose(f); + + return buf; +} + +/* ── esd_config_load ─────────────────────────────────────────────────── */ + +static void test_config_load_full_file(void) +{ + esd_config_t cfg; + int rc; + + memset(&cfg, 0, sizeof(cfg)); + + write_file(g_tmpdir, "esd.yaml", + "controller: 172.31.1.30:9990\n" + "poll_interval: 30\n" + "resources:\n" + " - name: Verilog_XL\n" + " collector: script\n" + " command: /etc/xce/lic-report.sh\n" + " interval: 15\n" + " - name: cloud_gpu_quota\n" + " collector: script\n" + " command: /etc/xce/quota-report.sh\n"); + + rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_STRING("172.31.1.30", cfg.controller_host); + TEST_ASSERT_EQUAL_INT(9990, cfg.controller_port); + TEST_ASSERT_EQUAL_INT(30, cfg.poll_interval); + TEST_ASSERT_EQUAL_INT(2, cfg.nsrc); + + TEST_ASSERT_NOT_NULL(cfg.src); + TEST_ASSERT_EQUAL_STRING("Verilog_XL", cfg.src[0].name); + TEST_ASSERT_EQUAL_STRING("/etc/xce/lic-report.sh", cfg.src[0].command); + TEST_ASSERT_EQUAL_INT(15, cfg.src[0].interval); + TEST_ASSERT_EQUAL_STRING("cloud_gpu_quota", cfg.src[1].name); + TEST_ASSERT_EQUAL_STRING("/etc/xce/quota-report.sh", cfg.src[1].command); + TEST_ASSERT_EQUAL_INT(0, cfg.src[1].interval); /* no override */ + + esd_config_fini(&cfg); +} + +static void test_config_load_missing_file_is_enoent(void) +{ + esd_config_t cfg; + int rc; + + memset(&cfg, 0, sizeof(cfg)); + + rc = esd_config_load_path(g_tmpdir, "no-such-file.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(-ENOENT, rc); +} + +static void test_config_load_bad_shape_is_einval(void) +{ + esd_config_t cfg; + int rc; + + memset(&cfg, 0, sizeof(cfg)); + + /* a source without a name is structurally invalid -- the controller + * would refuse the push and the operator would have no idea which + * entry died; fail loudly at load. */ + write_file(g_tmpdir, "esd.yaml", + "controller: 172.31.1.30:9990\n" + "resources:\n" + " - collector: script\n" + " command: /bin/true\n"); + + rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(-EINVAL, rc); + + /* a source without a command can never report -- same rule. */ + memset(&cfg, 0, sizeof(cfg)); + write_file(g_tmpdir, "esd.yaml", + "controller: 172.31.1.30:9990\n" + "resources:\n" + " - name: lic\n" + " collector: script\n"); + + rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(-EINVAL, rc); +} + +/* ── script collector output parsing (parse contract, not the spawn) ── */ + +static void test_parse_output_bare_number(void) +{ + double v = esd_parse_script_output("4\n"); + + TEST_ASSERT_FALSE(isnan(v)); + TEST_ASSERT_EQUAL_FLOAT((float)4.0, (float)v); +} + +static void test_parse_output_name_value(void) +{ + /* elim-style " " -- name may mismatch; the source it is + * configured for is authoritative, the name is decorative. */ + double v = esd_parse_script_output("Verilog_XL 7\n"); + + TEST_ASSERT_FALSE(isnan(v)); + TEST_ASSERT_EQUAL_FLOAT((float)7.0, (float)v); +} + +static void test_parse_output_whitespace_and_negative(void) +{ + double v = esd_parse_script_output(" \t 3.5 \n"); + + TEST_ASSERT_FALSE(isnan(v)); + TEST_ASSERT_EQUAL_FLOAT((float)3.5, (float)v); + + v = esd_parse_script_output("lic -1\n"); + TEST_ASSERT_FALSE(isnan(v)); + TEST_ASSERT_EQUAL_FLOAT((float)-1.0, (float)v); +} + +static void test_parse_output_garbage_is_nan(void) +{ + double v; + + v = esd_parse_script_output(""); + TEST_ASSERT_TRUE(isnan(v)); + v = esd_parse_script_output("\n"); + TEST_ASSERT_TRUE(isnan(v)); + v = esd_parse_script_output("no digits here\n"); + TEST_ASSERT_TRUE(isnan(v)); + v = esd_parse_script_output(NULL); + TEST_ASSERT_TRUE(isnan(v)); +} + +/* ── Phase 2: stale_limit config parsing ────────────────────────────── */ + +static void test_config_stale_limit_parse(void) +{ + esd_config_t cfg; + int rc; + + memset(&cfg, 0, sizeof(cfg)); + + /* per-source override wins; a source without one takes the daemon + * default; the daemon default itself is optional with a sane built-in + * (design.md 3.4: stale_limit in esd.yaml, fail-open keep-old-value). */ + write_file(g_tmpdir, "esd.yaml", + "controller: 172.31.1.30:9990\n" + "poll_interval: 10\n" + "stale_limit: 200\n" + "resources:\n" + " - name: lic_a\n" + " collector: script\n" + " command: /bin/a.sh\n" + " stale_limit: 60\n" + " - name: lic_b\n" + " collector: script\n" + " command: /bin/b.sh\n"); + + rc = esd_config_load_path(g_tmpdir, "esd.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(200, cfg.stale_limit); + TEST_ASSERT_EQUAL_INT(60, cfg.src[0].stale_limit); + TEST_ASSERT_EQUAL_INT(0, cfg.src[1].stale_limit); /* 0 = use default */ + + esd_config_fini(&cfg); +} + +/* ── Phase 2: stale predicate ───────────────────────────────────────── */ + +static void test_stale_predicate(void) +{ + esd_source_t src; + + memset(&src, 0, sizeof(src)); + src.name = (char*)"lic"; + src.have_value = true; + src.last_value = 3.0; + + /* Never report a value -> nothing to call stale; admission treats the + * pool as 0 through the normal NO_RESOURCE path. */ + src.last_ok_time = 0; + TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60)); + + /* Fresh success (10s ago, limit 60) -> not stale. */ + src.last_ok_time = 90; + TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60)); + + /* Exactly at the limit -> not stale (strictly-greater rule, so an + * interval == stale_limit config never flaps). */ + src.last_ok_time = 40; + TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60)); + + /* Past the limit -> stale. */ + src.last_ok_time = 39; + TEST_ASSERT_TRUE(esd_source_is_stale(&src, 100, 60)); + + /* A source whose own stale_limit override is set uses it. */ + src.stale_limit = 10; + TEST_ASSERT_TRUE(esd_source_is_stale(&src, 100, 60)); + src.last_ok_time = 95; + TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60)); + + /* last_ok_time newer than `now` (clock skew between test harness and + * caller) must not read as stale. */ + src.last_ok_time = 105; + TEST_ASSERT_FALSE(esd_source_is_stale(&src, 100, 60)); +} + +/* ── Phase 2: status file ───────────────────────────────────────────── */ + +static void test_status_write_basic(void) +{ + esd_config_t cfg; + esd_stats_t stats; + char path[512]; + char* text; + time_t now = time(NULL); + char expect_lvt[128]; + + memset(&cfg, 0, sizeof(cfg)); + cfg.nsrc = 1; + cfg.src = (esd_source_t*)calloc(1, sizeof(esd_source_t)); + TEST_ASSERT_NOT_NULL(cfg.src); + cfg.src[0].name = (char*)"extlic"; + cfg.src[0].have_value = true; + cfg.src[0].last_value = 3.0; + /* fresh success: last good value 5s ago, limit default 150 -> not + * stale (the marks-stale test below covers the other side) */ + cfg.src[0].last_ok_time = now - 5; + cfg.stale_limit = ESD_STALE_LIMIT_DEFAULT; + + memset(&stats, 0, sizeof(stats)); + stats.started_at = 1000; + stats.pushes_ok = 7; + stats.pushes_fail = 1; + + snprintf(path, sizeof(path), "%s/status.json", g_tmpdir); + TEST_ASSERT_EQUAL_INT(0, esd_status_write(path, &cfg, &stats)); + + text = slurp(path); + TEST_ASSERT_NOT_NULL(text); + + /* The fields xceadmin extres status reads, and the stale flag driven + * by the same predicate the tests above pin. */ + TEST_ASSERT_TRUE(strstr(text, "\"daemon\": \"xceesd\"") != NULL); + TEST_ASSERT_TRUE(strstr(text, "\"name\": \"extlic\"") != NULL); + snprintf(expect_lvt, sizeof(expect_lvt), "\"last_value\": %.17g", 3.0); + TEST_ASSERT_TRUE(strstr(text, expect_lvt) != NULL); + TEST_ASSERT_TRUE(strstr(text, "\"last_ok_time\": ") != NULL); + TEST_ASSERT_TRUE(strstr(text, "\"stale\": false") != NULL); + TEST_ASSERT_TRUE(strstr(text, "\"pushes_ok\": 7") != NULL); + TEST_ASSERT_TRUE(strstr(text, "\"pushes_fail\": 1") != NULL); + + free(text); + free(cfg.src); +} + +static void test_status_write_marks_stale(void) +{ + esd_config_t cfg; + esd_stats_t stats; + char path[512]; + char* text; + time_t now = time(NULL); + + memset(&cfg, 0, sizeof(cfg)); + cfg.nsrc = 1; + cfg.src = (esd_source_t*)calloc(1, sizeof(esd_source_t)); + TEST_ASSERT_NOT_NULL(cfg.src); + cfg.src[0].name = (char*)"extlic"; + cfg.src[0].have_value = true; + cfg.src[0].last_value = 2.0; + /* last success 1 hour ago, limit 150s -> stale. */ + cfg.src[0].last_ok_time = now - 3600; + cfg.src[0].stale_limit = 150; + + memset(&stats, 0, sizeof(stats)); + + snprintf(path, sizeof(path), "%s/status.json", g_tmpdir); + TEST_ASSERT_EQUAL_INT(0, esd_status_write(path, &cfg, &stats)); + + text = slurp(path); + TEST_ASSERT_NOT_NULL(text); + TEST_ASSERT_TRUE(strstr(text, "\"stale\": true") != NULL); + + free(text); + free(cfg.src); +} + +/* ── bug-1978: the reconnect-flag contract ───────────────────────────── */ + +static void test_reconnect_sets_both_rejoin_and_full_push(void) +{ + g_needs_rejoin = 0; + g_needs_full_push = 0; + + TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x1, NULL)); + + /* BOTH flags: the re-join must happen before any re-push (an un-joined + * push is refused -EPERM on the fresh endpoint), and the re-push must + * still happen (the fresh controller knows no values). The bug-1978 + * defect cleared only g_needs_full_push -- a daemon that re-pushed onto + * an un-joined HostRole_client endpoint, every push -EPERM forever. */ + TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin); + TEST_ASSERT_EQUAL_INT(1, g_needs_full_push); +} + +static void test_reconnect_flags_recover_from_any_previous_state(void) +{ + /* The handler is level-set, not level-toggle: regardless of what a + * previous (already consumed) reconnect left behind, a new callback + * invocation re-arms both flags. */ + g_needs_rejoin = 0; + g_needs_full_push = 1; /* stale leftover from an unconsumed cycle */ + + TEST_ASSERT_EQUAL_INT(0, esd_on_reconnect((xcecli_ctx_t)0x2, NULL)); + + TEST_ASSERT_EQUAL_INT(1, g_needs_rejoin); + TEST_ASSERT_EQUAL_INT(1, g_needs_full_push); +} + +int main(void) +{ + UNITY_BEGIN(); + + RUN_TEST(test_config_load_full_file); + RUN_TEST(test_config_load_missing_file_is_enoent); + RUN_TEST(test_config_load_bad_shape_is_einval); + RUN_TEST(test_parse_output_bare_number); + RUN_TEST(test_parse_output_name_value); + RUN_TEST(test_parse_output_whitespace_and_negative); + RUN_TEST(test_parse_output_garbage_is_nan); + RUN_TEST(test_config_stale_limit_parse); + RUN_TEST(test_stale_predicate); + RUN_TEST(test_status_write_basic); + RUN_TEST(test_status_write_marks_stale); + RUN_TEST(test_reconnect_sets_both_rejoin_and_full_push); + RUN_TEST(test_reconnect_flags_recover_from_any_previous_state); + + return UNITY_END(); +} diff --git a/src/frontend/apiserver/backend/convert.go b/src/frontend/apiserver/backend/convert.go index 5df2b2af..0993535b 100644 --- a/src/frontend/apiserver/backend/convert.go +++ b/src/frontend/apiserver/backend/convert.go @@ -286,12 +286,13 @@ func cQueueToModel(queue *C.xce_queue_t) *models.QueueInfo { // cResourceToModel extracts C xce_resource_t fields and maps them to ResourceDefinitionInfo. func cResourceToModel(res *C.xce_resource_t) *models.ResourceDefinitionInfo { return rawResourceToModel(rawResourceData{ - name: goString(res.name), - desc: goString(res.desc), - valType: int(res.valtype), - ordType: int(res.ordtype), - flags: int(res.flags), - interval: int(res.interval), + name: goString(res.name), + desc: goString(res.desc), + valType: int(res.valtype), + ordType: int(res.ordtype), + flags: int(res.flags), + interval: int(res.interval), + lastReport: int64(res.last_report), }) } diff --git a/src/frontend/apiserver/backend/convert_pure.go b/src/frontend/apiserver/backend/convert_pure.go index feb0cf8d..5c6a26f7 100644 --- a/src/frontend/apiserver/backend/convert_pure.go +++ b/src/frontend/apiserver/backend/convert_pure.go @@ -578,12 +578,13 @@ type rawQueueData struct { // rawResourceData mirrors the C xce_resource_t fields used by the Go model. type rawResourceData struct { - name string - desc string - valType int - ordType int - flags int - interval int + name string + desc string + valType int + ordType int + flags int + interval int + lastReport int64 } // interactiveURLFromExdata extracts the interactive session URL from a job @@ -828,6 +829,7 @@ func rawResourceToModel(r rawResourceData) *models.ResourceDefinitionInfo { OrderType: r.ordType, Flag: r.flags, Interval: r.interval, + LastReport: r.lastReport, } } diff --git a/src/frontend/apiserver/backend/handler.go b/src/frontend/apiserver/backend/handler.go index d484f839..bd6140fe 100644 --- a/src/frontend/apiserver/backend/handler.go +++ b/src/frontend/apiserver/backend/handler.go @@ -5,6 +5,7 @@ import ( "apiserver/store" logger "apiserver/utils" "sync" + "time" "go.uber.org/zap" ) @@ -249,6 +250,23 @@ func (h *DefaultHandler) OnResourceLocationUpdated(res *models.ResourceDistribut key := res.ResourceName + ":" + res.Location store.ResourceLocations.Put(key, *res) + // external-license-resources Phase 2: the definition store is fed ONLY + // by the join-time config push, whose snapshot carries whatever the + // config tree had at push time -- so an EXTERNAL pool's last_report + // would stay 0 ("never") for the whole apiserver process lifetime even + // while the extres daemon pushes normally. A shared-value push (the + // daemon's landing shape: Location "") for an EXTERNAL definition is + // exactly an accepted external report at the controller, so refresh the + // stamp here. Host-sourced values (Location != "") and non-EXTERNAL + // resources must not touch it: last_report means "external value last + // accepted", and worker reports are a different freshness domain. + if res.Location == "" { + if def, ok := store.Resources.Get(res.ResourceName); ok && (def.Flag&models.EXTERNAL) != 0 { + def.LastReport = time.Now().Unix() + store.Resources.Put(def.ResourceName, def) + } + } + // resource-value-verification task 4.2 (Requirements 5.1, 5.2, 5.4): trace // the stored value at its point of application, mirroring the scheduler's // sibling trace log added in task 4.1 (core_events.c). One typed "value" diff --git a/src/frontend/apiserver/backend/integration_test.go b/src/frontend/apiserver/backend/integration_test.go index baa1cd98..ef5ac6fb 100644 --- a/src/frontend/apiserver/backend/integration_test.go +++ b/src/frontend/apiserver/backend/integration_test.go @@ -8,6 +8,7 @@ import ( "apiserver/utils" "sync/atomic" "testing" + "time" "go.uber.org/zap" ) @@ -516,3 +517,71 @@ func TestIntegration_CrossFeature_SequentialConfigPushes(t *testing.T) { t.Errorf("heartbeatIntervalSecs = %d, want 60", hb) } } + +// --- 5.6 Phase 2 (external-license-resources): a shared-value push refreshes +// the stored EXTERNAL definition's last_report. The definition store is fed +// only by the join-time config push, whose snapshot carries whatever the +// config tree had at push time — without this refresh the CLI's LAST_UPDATED +// column would show "never" for the whole apiserver process lifetime even +// while the extres daemon pushes normally. The daemon landing shape for a +// cluster-level value is Location=="" (host:"" on the wire). Worker-sourced +// host values must NOT touch the stamp. --- +func TestIntegration_ExternalSharedPushRefreshesDefinitionLastReport(t *testing.T) { + resetIntegrationStores() + h := &DefaultHandler{} + + // config push seeds the definition store: extlic is EXTERNAL (0x10), + // mem is a plain host-based resource. + dispatchConfigPush(h, nil, []models.ResourceDefinitionInfo{ + {ResourceName: "extlic", Flag: 0x80 | 0x10, LastReport: 0}, + {ResourceName: "mem", Flag: 0x80, LastReport: 0}, + }, nil, &models.ClusterInfo{ClusterName: "c1"}) + + before := time.Now().Unix() + + // daemon-shaped shared push for extlic (Location "") + h.OnResourceLocationUpdated(&models.ResourceDistributionInfo{ + ResourceName: "extlic", Location: "", NumericValue: 4.0, + ValueType: int(models.NUMERIC), IsDynamic: true, + }) + + got, ok := store.Resources.Get("extlic") + if !ok { + t.Fatal("extlic definition should exist after config push") + } + if got.LastReport < before { + t.Errorf("extlic LastReport = %d, want >= %d (refreshed by the shared push)", got.LastReport, before) + } + + // host-sourced push for mem must not stamp its definition + h.OnResourceLocationUpdated(&models.ResourceDistributionInfo{ + ResourceName: "mem", Location: "node01", NumericValue: 65536.0, + ValueType: int(models.NUMERIC), IsDynamic: true, + }) + gotMem, _ := store.Resources.Get("mem") + if gotMem.LastReport != 0 { + t.Errorf("mem LastReport = %d, want 0 (host values never stamp)", gotMem.LastReport) + } +} + +// The refresh only applies to EXTERNAL definitions: a shared push for a +// non-EXTERNAL resource (e.g. a dynamic shared pool reported by workers) +// leaves last_report untouched. +func TestIntegration_SharedPushDoesNotStampNonExternalDefinition(t *testing.T) { + resetIntegrationStores() + h := &DefaultHandler{} + + dispatchConfigPush(h, nil, []models.ResourceDefinitionInfo{ + {ResourceName: "plainshared", Flag: 0x80 | 0x08, LastReport: 0}, + }, nil, &models.ClusterInfo{ClusterName: "c1"}) + + h.OnResourceLocationUpdated(&models.ResourceDistributionInfo{ + ResourceName: "plainshared", Location: "", NumericValue: 2.0, + ValueType: int(models.NUMERIC), IsDynamic: true, + }) + + got, _ := store.Resources.Get("plainshared") + if got.LastReport != 0 { + t.Errorf("plainshared LastReport = %d, want 0 (not EXTERNAL)", got.LastReport) + } +} diff --git a/src/frontend/apiserver/models/resources.go b/src/frontend/apiserver/models/resources.go index e6afde0c..dca0441d 100644 --- a/src/frontend/apiserver/models/resources.go +++ b/src/frontend/apiserver/models/resources.go @@ -67,6 +67,10 @@ var RESOURCE_DEFINITION_DEFAULT_COLUMN = []string{ "order_type", "flag", "interval", + // external-license-resources Phase 2: SelectFields drops row keys not + // listed here, so the external value's last-report timestamp must be a + // default column or it never reaches API clients / the CLI. + "last_report", } var ResourceFieldMappings = map[string]string{ @@ -77,8 +81,12 @@ var ResourceFieldMappings = map[string]string{ var ResourcesAvailableTypes = []string{"static", "dynamic"} const ( - BUILTIN = 0x01 - DYNAMIC = 0x02 + BUILTIN = 0x01 + DYNAMIC = 0x02 + // external-license-resources: EXTERNAL=0x10 mirrors C's XCE_RESF_EXTERNAL + // (src/include/xce/types.h) — the value comes from an external daemon, + // not from workers. + EXTERNAL = 0x10 RELEASE = 0x20 CONSUMABLE = 0x80 ) @@ -132,6 +140,9 @@ type ResourceDefinitionInfo struct { OrderType int `json:"order_type"` Flag int `json:"flag"` Interval int `json:"interval"` + // external-license-resources Phase 2: epoch seconds the external value + // was last accepted by the controller (0 = never / not external). + LastReport int64 `json:"last_report"` } var RESOURCE_DISTRIBUTION_SUPPORT_SORT = []string{ diff --git a/src/frontend/apiserver/models/resources_test.go b/src/frontend/apiserver/models/resources_test.go new file mode 100644 index 00000000..175d1051 --- /dev/null +++ b/src/frontend/apiserver/models/resources_test.go @@ -0,0 +1,47 @@ +package models + +import ( + "encoding/json" + "testing" +) + +// external-license-resources Phase 2: last_report must be part of the default +// column list for /resources/definition — SelectFields drops any row key not +// listed there, so without it the API response (and the CLI LAST_UPDATED +// column) never sees the timestamp even though the model carries it. +func TestResourceDefinitionDefaultColumnIncludesLastReport(t *testing.T) { + found := false + for _, col := range RESOURCE_DEFINITION_DEFAULT_COLUMN { + if col == "last_report" { + found = true + break + } + } + if !found { + t.Errorf("RESOURCE_DEFINITION_DEFAULT_COLUMN = %v, missing \"last_report\"", RESOURCE_DEFINITION_DEFAULT_COLUMN) + } +} + +// ResourceDefinitionInfo must serialize last_report with the json tag the CLI +// model (xcecli/models/resource.go) unmarshals. +func TestResourceDefinitionInfoJSONCarriesLastReport(t *testing.T) { + info := ResourceDefinitionInfo{ + ResourceName: "extlic", + LastReport: 1800000000, + } + data, err := json.Marshal(info) + if err != nil { + t.Fatalf("json.Marshal: %v", err) + } + var m map[string]interface{} + if err := json.Unmarshal(data, &m); err != nil { + t.Fatalf("json.Unmarshal: %v", err) + } + v, ok := m["last_report"] + if !ok { + t.Fatalf("serialized body %s has no last_report key", data) + } + if v.(float64) != 1800000000 { + t.Errorf("last_report = %v, want 1800000000", v) + } +} diff --git a/src/frontend/xcecli/cmd/jobs/submit.go b/src/frontend/xcecli/cmd/jobs/submit.go index 1da4a801..86134c46 100755 --- a/src/frontend/xcecli/cmd/jobs/submit.go +++ b/src/frontend/xcecli/cmd/jobs/submit.go @@ -896,10 +896,20 @@ func (s *stringValue) Set(val string) error { func (s *stringValue) Type() string { return "string" } func (js *JobSubmitter) getEnvironmentVariables() []string { + // The downstream wire path (apiserver ConvertStringToJsonb) splits each + // entry at the FIRST '=' (strings.SplitN 2), so values containing '=' + // (A=b=c -> A = "b=c") round-trip fine -- dropping them here used to be + // a silent data loss, not a necessary restriction. A nameless entry + // (=value) cannot become a map key, so it is the one case that must be + // rejected -- say so instead of dropping it silently. var envVars []string for _, env := range js.subEnvVars { - // Skip environment variables with multiple equal signs - if strings.Count(env, "=") > 1 { + if strings.HasPrefix(env, "=") { + fmt.Printf("warning: ignoring environment variable '%s': empty name -- name=value entries only\n", env) + continue + } + if !strings.Contains(env, "=") { + fmt.Printf("warning: ignoring environment variable '%s': no '=' -- name=value entries only\n", env) continue } envVars = append(envVars, env) diff --git a/src/frontend/xcecli/cmd/jobs/submit_test.go b/src/frontend/xcecli/cmd/jobs/submit_test.go index 3ccf2e74..d2ac9466 100644 --- a/src/frontend/xcecli/cmd/jobs/submit_test.go +++ b/src/frontend/xcecli/cmd/jobs/submit_test.go @@ -1,8 +1,10 @@ package jobs import ( + "bytes" "encoding/json" "fmt" + "io" "net/http" "net/http/httptest" "os" @@ -133,6 +135,48 @@ func TestSubmitHelpShowsArrayForms(t *testing.T) { } } +// getEnvironmentVariables must keep entries whose value contains '=' (the +// apiserver splits at the FIRST '=', so A=b=c is A -> "b=c") -- dropping +// them used to be silent data loss. A nameless or '='-less entry cannot +// become a map key, so it is warned about and dropped. +func TestGetEnvironmentVariables_KeepsMultiEqualSign(t *testing.T) { + js := NewJobSubmitter(nil) + js.subEnvVars = []string{"A=1", "B=b=c", "C=", "=novalue", "novalue"} + + r, w, pipeErr := os.Pipe() + if pipeErr != nil { + t.Fatalf("pipe: %v", pipeErr) + } + old := os.Stdout + os.Stdout = w + done := make(chan string) + go func() { + var buf bytes.Buffer + _, _ = io.Copy(&buf, r) + done <- buf.String() + }() + got := js.getEnvironmentVariables() + os.Stdout = old + _ = w.Close() + warn := <-done + + want := []string{"A=1", "B=b=c", "C="} + if len(got) != len(want) { + t.Fatalf("env = %q, want %q", got, want) + } + for i := range want { + if got[i] != want[i] { + t.Errorf("env[%d] = %q, want %q", i, got[i], want[i]) + } + } + if !strings.Contains(warn, "=novalue") || !strings.Contains(warn, "warning:") { + t.Errorf("warning output = %q, want it to name the nameless entry", warn) + } + if !strings.Contains(warn, "novalue") { + t.Errorf("warning output = %q, want it to name the '='-less entry", warn) + } +} + // Attach marks the terminal-attach kind of TTY job so the worker may reap // ttyd when no client ever connects; --browse keeps the web-viewed // semantics and must not set it. diff --git a/src/frontend/xcecli/common/output/page_print_resources.go b/src/frontend/xcecli/common/output/page_print_resources.go index 32469cf4..eb9386a8 100644 --- a/src/frontend/xcecli/common/output/page_print_resources.go +++ b/src/frontend/xcecli/common/output/page_print_resources.go @@ -3,6 +3,7 @@ package output import ( "fmt" "strings" + "time" "xcecli/models" ) @@ -143,6 +144,8 @@ func (p *ResourcesPagePrinter) initColumnWidthsForDef(resourcesDef *models.Resou width = len(resourceDef.ResourceType) case models.HEADER_RESOURCE_DEF_DESC: width = len(resourceDef.Description) + case models.HEADER_RESOURCE_DEF_LAST_REPORT: + width = len(formatJobTime(time.Unix(resourceDef.LastReport, 0))) } if !p.wideFormat && width > 15 { @@ -270,6 +273,8 @@ func (p *ResourcesPagePrinter) formatRowsForDef(resourcesDef *models.ResourceDef row = appendOrDefault(row, FormatStringField(builtinValue, width, truncatFlag, true), "-") case models.HEADER_RESOURCE_DEF_INTERVAL: row = appendOrDefault(row, fmt.Sprintf("%"+align+"*d", width, resourceDef.Interval), "0") + case models.HEADER_RESOURCE_DEF_LAST_REPORT: + row = appendOrDefault(row, FormatStringField(fmtUnixTime(resourceDef.LastReport), width, truncatFlag, true), "-") } } rows = append(rows, row) @@ -354,6 +359,8 @@ func printResourcesDefVerboseOutput(resourceDef *models.ResourceDefResponse) { val = releaseValue case models.HEADER_RESOURCE_DEF_INTERVAL: val = fmt.Sprintf("%d", resourceDef.Interval) + case models.HEADER_RESOURCE_DEF_LAST_REPORT: + val = fmtUnixTime(resourceDef.LastReport) } widths1[i] = max(len(h), len(val)) row1 = append(row1, FormatStringField(val, widths1[i], false, true)) diff --git a/src/frontend/xcecli/models/resource.go b/src/frontend/xcecli/models/resource.go index cd72068f..34927b8d 100644 --- a/src/frontend/xcecli/models/resource.go +++ b/src/frontend/xcecli/models/resource.go @@ -7,17 +7,18 @@ import ( ) const ( - HEADER_RESOURCE_NAME = "RESOURCE" - HEADER_RESOURCE_VALUE = "VALUE" - HEADER_RESOURCE_LOC = "LOCATION" - HEADER_RESOURCE_DEF_TYPE = "TYPE" - HEADER_RESOURCE_DEF_ORDER = "ORDER" - HEADER_RESOURCE_DEF_BUILTIN = "BUILTIN" - HEADER_RESOURCE_DEF_CONSUMABLE = "CONSUMABLE" - HEADER_RESOURCE_DEF_DYNAMIC = "DYNAMIC" - HEADER_RESOURCE_DEF_RELEASE = "RELEASE" - HEADER_RESOURCE_DEF_INTERVAL = "INTERVAL" - HEADER_RESOURCE_DEF_DESC = "DESCRIPTION" + HEADER_RESOURCE_NAME = "RESOURCE" + HEADER_RESOURCE_VALUE = "VALUE" + HEADER_RESOURCE_LOC = "LOCATION" + HEADER_RESOURCE_DEF_TYPE = "TYPE" + HEADER_RESOURCE_DEF_ORDER = "ORDER" + HEADER_RESOURCE_DEF_BUILTIN = "BUILTIN" + HEADER_RESOURCE_DEF_CONSUMABLE = "CONSUMABLE" + HEADER_RESOURCE_DEF_DYNAMIC = "DYNAMIC" + HEADER_RESOURCE_DEF_RELEASE = "RELEASE" + HEADER_RESOURCE_DEF_INTERVAL = "INTERVAL" + HEADER_RESOURCE_DEF_LAST_REPORT = "LAST_UPDATED" + HEADER_RESOURCE_DEF_DESC = "DESCRIPTION" ) var ResourceDistFieldsMap = map[string]string{ @@ -28,21 +29,23 @@ var ResourceDistFieldsMap = map[string]string{ // ResourceDefFieldsMap maps resource definition field headers to API field names var ResourceDefFieldsMap = map[string]string{ - HEADER_RESOURCE_NAME: "res_name", - HEADER_RESOURCE_DEF_DESC: "description", - HEADER_RESOURCE_DEF_TYPE: "type", - HEADER_RESOURCE_DEF_ORDER: "order", - HEADER_RESOURCE_DEF_BUILTIN: "builtin", - HEADER_RESOURCE_DEF_CONSUMABLE: "consumable", - HEADER_RESOURCE_DEF_DYNAMIC: "dynamic", - HEADER_RESOURCE_DEF_RELEASE: "release", - HEADER_RESOURCE_DEF_INTERVAL: "interval", + HEADER_RESOURCE_NAME: "res_name", + HEADER_RESOURCE_DEF_DESC: "description", + HEADER_RESOURCE_DEF_TYPE: "type", + HEADER_RESOURCE_DEF_ORDER: "order", + HEADER_RESOURCE_DEF_BUILTIN: "builtin", + HEADER_RESOURCE_DEF_CONSUMABLE: "consumable", + HEADER_RESOURCE_DEF_DYNAMIC: "dynamic", + HEADER_RESOURCE_DEF_RELEASE: "release", + HEADER_RESOURCE_DEF_INTERVAL: "interval", + HEADER_RESOURCE_DEF_LAST_REPORT: "last_report", } var ResourceDefDetailHeaders = []string{ HEADER_RESOURCE_DEF_TYPE, HEADER_RESOURCE_DEF_ORDER, HEADER_RESOURCE_DEF_INTERVAL, + HEADER_RESOURCE_DEF_LAST_REPORT, HEADER_RESOURCE_DEF_BUILTIN, HEADER_RESOURCE_DEF_DYNAMIC, HEADER_RESOURCE_DEF_RELEASE, @@ -177,6 +180,7 @@ type ResourceDef struct { OrderType string `json:"-"` // populated from order_type int Flag map[string]interface{} `json:"flag"` Interval int `json:"interval"` + LastReport int64 `json:"last_report"` // Unix seconds of the last external value report (0 = never) } func (r *ResourceDef) UnmarshalJSON(data []byte) error { @@ -187,6 +191,7 @@ func (r *ResourceDef) UnmarshalJSON(data []byte) error { OrderType string `json:"order_type"` Flag map[string]interface{} `json:"flag"` Interval int `json:"interval"` + LastReport int64 `json:"last_report"` } var a Alias if err := UnmarshalOrderMapRow(data, &a); err != nil { @@ -198,5 +203,6 @@ func (r *ResourceDef) UnmarshalJSON(data []byte) error { r.OrderType = a.OrderType r.Flag = a.Flag r.Interval = a.Interval + r.LastReport = a.LastReport return nil } diff --git a/src/include/xce/cluster_def.h b/src/include/xce/cluster_def.h index 5397d63d..1a337279 100644 --- a/src/include/xce/cluster_def.h +++ b/src/include/xce/cluster_def.h @@ -27,6 +27,7 @@ #define N_CONTROLLER_MAX (16) #define N_APISERVER_MAX (16) +#define N_EXTRES_MAX (8) enum { XCE_CONTROLLER_HOOK_JOB = 0, @@ -96,6 +97,18 @@ typedef struct { endpoint_t* ep; } cluster_apiserver_t; +/* external-license-resources Phase 1: connected extres daemons (xceesd). + * Registered at HostRole_extres join like the apiserver table -- endpoint + + * name only, no workermgr slot (the daemon is not a schedulable host). The + * controller never pushes TO the daemon (the daemon pulls nothing and + * re-pushes its full value set on every reconnect), so this table exists + * for leave-cleanup and future admin surface (xceadmin extres status). */ +typedef struct { + mylock_t lock; + char* name; + endpoint_t* ep; +} cluster_extres_t; + typedef struct cluster_s { myconf_t* myconf; xce_peerinfo_t* myhost; /* points at myhost_info once initialised */ @@ -108,6 +121,7 @@ typedef struct cluster_s { zarray_t sharedinst; controller_t controllers[N_CONTROLLER_MAX]; cluster_apiserver_t apiservers[N_APISERVER_MAX]; + cluster_extres_t extresds[N_EXTRES_MAX]; int myidx; int master; int ncontrollers; diff --git a/src/lib/request.c b/src/lib/request.c index 92902955..e94490e9 100644 --- a/src/lib/request.c +++ b/src/lib/request.c @@ -1414,7 +1414,8 @@ int xce_join_req(endpoint_t* ep, int role, hostattrs_t* attrs) if ((role != HostRole_client) && (role != HostRole_worker) && (role != HostRole_apiserver) && - (role != HostRole_scheduler)) { + (role != HostRole_scheduler) && + (role != HostRole_extres)) { return -EINVAL; } /* clang-format on */ diff --git a/src/lib/resreq.c b/src/lib/resreq.c index fad1e46a..1ee31b70 100644 --- a/src/lib/resreq.c +++ b/src/lib/resreq.c @@ -473,6 +473,23 @@ int parse_resreq_primary(xce_primary_resreq_t* primary, char* resreqString, if (data.selectString != NULL && data.selectString[0] != '\0') { rc = parse_select(primary, data.selectString, resmgr, errmsg); if (rc < 0) { + /* -ENOENT (select names an unregistered resource) is tolerated + * by the resreq grammar -- xce_resreq.peg keeps the primary + * without the clause (jobs rely on it, the scheduler's own + * "type == any" default among them when no "type" resource is + * registered). That tolerance is silent, and the early return + * below also skips every clause that follows in this primary, + * so say so: a job whose select and rusage are both gone is + * scheduled with none of the constraints it asked for. + * -EINVAL/-ENOMEM failures need no WARN here -- the grammar + * turns them into a whole-resreq rejection the caller logs. */ + if (rc == -ENOENT) { + WARN("resreq: select '%s' references an unregistered " + "resource -- dropping the select and skipping any " + "rusage[] clause of this primary; the job is scheduled " + "without these constraints", + data.selectString); + } kill_resreq_data(&data); return rc; } diff --git a/src/lib/testsuite/test_select.c b/src/lib/testsuite/test_select.c index 35dbd914..df299f69 100644 --- a/src/lib/testsuite/test_select.c +++ b/src/lib/testsuite/test_select.c @@ -128,6 +128,33 @@ test_select_parse_unknown_resource(void) sel_optimize_test_reset(); } +static void +test_select_unknown_resource_skips_rusage(void) +{ + /* The -ENOENT tolerance (bug-1887) drops the select AND the primary's + * rusage clause with it (parse_resreq_primary returns before parsing + * rusage) -- now with a WARN. Assert the documented drop semantics: + * rc carries -ENOENT and no rusage survived, so callers can never + * mistake the tolerated failure for a fully-parsed primary. */ + char * errmsg = NULL; + xce_primary_resreq_t resreq; + + var resmgr = make_test_resmgr(); + + sel_optimize_set(0); + + int rc = parse_resreq_primary(&resreq, "unknownres==linux rusage[mem=2]", + resmgr, &errmsg); + TEST_ASSERT_EQUAL_INT(-ENOENT, rc); + TEST_ASSERT_NULL(resreq.select); + TEST_ASSERT_EQUAL_INT(0, resreq.rusages.count); + + kill_primary_resreq(&resreq); + FREEUP(errmsg); + free_test_resmgr(resmgr); + sel_optimize_test_reset(); +} + static void test_select_parse_type_mismatch(void) { @@ -2346,6 +2373,7 @@ test_select(void) RUN_TEST(test_select_parse_valid_expr); RUN_TEST(test_select_parse_valid_any); RUN_TEST(test_select_parse_unknown_resource); + RUN_TEST(test_select_unknown_resource_skips_rusage); RUN_TEST(test_select_parse_type_mismatch); RUN_TEST(test_select_input_len_overflow); diff --git a/src/meson.build b/src/meson.build index 6d3b0a34..a511b477 100644 --- a/src/meson.build +++ b/src/meson.build @@ -76,6 +76,7 @@ if component != 'xcecli' subdir('controller') subdir('worker') subdir('autoscaler') + subdir('esd') subdir('scheduler') subdir('frontend/apiserver') subdir('xceadmin') @@ -88,7 +89,7 @@ if component != 'xcecli' find_program('python3'), args: [meson.project_source_root() / '..' / 'scripts' / 'check-no-fork.py', meson.project_source_root() / '..'], - suite: ['lib', 'controller', 'scheduler', 'worker']) + suite: ['lib', 'controller', 'scheduler', 'worker', 'esd']) configure_file( input: 'tools/appimage/entry.sh', diff --git a/src/scheduler/src/core_events.c b/src/scheduler/src/core_events.c index 156241a5..47a4caec 100644 --- a/src/scheduler/src/core_events.c +++ b/src/scheduler/src/core_events.c @@ -1184,10 +1184,16 @@ int xce_core_push_events(xce_sched_core_t* core, xce_core_event_t* events, if ((key_find(core->resmgr, res->name, (void**)&found) == 0) && (found != NULL)) { - found->valtype = res->valtype; - found->ordtype = res->ordtype; - found->flags = res->flags; - found->interval = res->interval; + found->valtype = res->valtype; + found->ordtype = res->ordtype; + found->flags = res->flags; + found->interval = res->interval; + /* external-license-resources Phase 2: the extres + * freshness stamp (controller stamps it at each + * accepted push); resobj.c's makeResource excludes + * last_report from definition comparison, so this + * is metadata, not an identity change. */ + found->last_report = res->last_report; } else { WARN("%s RESOURCES_PUSH: makeResource failed for " "%s and lookup not found", @@ -1234,16 +1240,27 @@ int xce_core_push_events(xce_sched_core_t* core, xce_core_event_t* events, } for (r = 0; r < deltas->count; r++) { - xce_resval_t* rv = deltas->vals[r]; - xce_sched_worker_t* w = NULL; - xce_resource_t* res = NULL; + xce_resval_t* rv = deltas->vals[r]; + xce_sched_worker_t* w = NULL; + xce_resource_t* res = NULL; + int is_shared; if (rv == NULL) { continue; } - w = sched_get_worker(core, rv->host); - if (w == NULL) { + /* external-license-resources Phase 1: a NULL/"" host marks a + * SHARED (cluster-level) value -- the extres daemon pushes + * license free counts this way (worker pushes always name a + * host). Land it in core->sharedInst, mirroring + * snapshot_load()'s host==NULL/"" precedent (bug-1923). + * Before this rule the empty host went through + * sched_get_worker() -> NULL -> WARN-skip, so daemon-pushed + * shared values never reached the scheduler. */ + is_shared = (rv->host == NULL) || (rv->host[0] == '\0'); + + w = is_shared ? NULL : sched_get_worker(core, rv->host); + if (!is_shared && (w == NULL)) { WARN("%s RESVALS_DELTA: unknown worker %s, skipping", __func__, rv->host); continue; @@ -1256,6 +1273,19 @@ int xce_core_push_events(xce_sched_core_t* core, xce_core_event_t* events, continue; } + if (is_shared) { + /* bug-1585 ownership rule: dup_val(), not a shallow + * rv->val store -- same hazard as the per-worker store + * below. */ + zarray_put(&(core->sharedInst), res->idx, dup_val(rv->val)); + if ((rv->val != NULL) + && (rv->val->which == Value_u_fvalue)) { + TRACE("resval applied host=%s resource=%s value=%f", + "(shared)", res->name, rv->val->u.fvalue); + } + continue; + } + /* bug-1585: dup_val(), not a shallow `rv->val` store -- see * this file's own top-of-file comment on the `dup_val` * extern declaration for the full ownership hazard this diff --git a/src/scheduler/src/resource.c b/src/scheduler/src/resource.c index a8945126..0cce8a42 100644 --- a/src/scheduler/src/resource.c +++ b/src/scheduler/src/resource.c @@ -159,11 +159,26 @@ static int unsatisfy_(xce_sched_core_t* core, int nres, int* idxs, float* total, count = 0; for (i = 0; i < nres; i++) { - float uncomit = uncomit_at_(uncomitted, idxs[i]); + float uncomit = uncomit_at_(uncomitted, idxs[i]); + xce_resource_t* r = NULL; + int is_external; - if (total[i] < inuse[i] + uncomit + wanted[i]) { - TRACE("%s resource %d total %f inuse %f uncomit %f wanted %f", - __func__, idxs[i], total[i], inuse[i], uncomit, wanted[i]); + /* external-license-resources: an EXTERNAL resource's pushed value + * is already a FREE count (license server's remaining, net of both + * external consumers and our own checked-out jobs), so re-subtracting + * the scheduler's own usage bookkeeping (sharedInUse / w->resInUse) + * would count our usage twice. Skip the inuse term for these: + * EXTERNAL: total >= uncomit + wanted + * non-EXTERNAL: total >= inuse + uncomit + wanted + * (r->flags is fetched here for the trace line's resource name too -- + * the flag was already reachable in this loop's vget). */ + vget(core->resmgr, idxs[i], (void**)&r); + is_external = (r != NULL) && (r->flags & XCE_RESF_EXTERNAL); + + if (total[i] < ((is_external ? 0.0f : inuse[i]) + uncomit + wanted[i])) { + TRACE("%s resource %d total %f inuse %f uncomit %f wanted %f%s", + __func__, idxs[i], total[i], is_external ? 0.0f : inuse[i], + uncomit, wanted[i], is_external ? " (external, inuse skipped)" : ""); count++; } @@ -175,13 +190,17 @@ static int unsatisfy_(xce_sched_core_t* core, int nres, int* idxs, float* total, j = 0; for (i = 0; i < nres; i++) { - float uncomit = uncomit_at_(uncomitted, idxs[i]); + float uncomit = uncomit_at_(uncomitted, idxs[i]); + xce_resource_t* r = NULL; + int is_external; - if (total[i] < inuse[i] + uncomit + wanted[i]) { - xce_resource_t* r = NULL; - float need = inuse[i] + uncomit + wanted[i] - total[i]; + vget(core->resmgr, idxs[i], (void**)&r); + is_external = (r != NULL) && (r->flags & XCE_RESF_EXTERNAL); + + if (total[i] < ((is_external ? 0.0f : inuse[i]) + uncomit + wanted[i])) { + float need = (is_external ? 0.0f : inuse[i]) + uncomit + wanted[i] + - total[i]; - vget(core->resmgr, idxs[i], (void**)&r); TRACE("%s unmet resource %d (%s) shortfall %f", __func__, idxs[i], r ? r->name : "?", need); diff --git a/src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay b/src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay index a79b84a1..8e04013d 100644 Binary files a/src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay and b/src/scheduler/testsuite/fixtures/repl_trace_baseline/contention.replay differ diff --git a/src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay b/src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay index 70293245..c3aa4d51 100644 Binary files a/src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay and b/src/scheduler/testsuite/fixtures/repl_trace_baseline/set_output.replay differ diff --git a/src/scheduler/testsuite/fixtures/repl_trace_baseline/sim_output.replay b/src/scheduler/testsuite/fixtures/repl_trace_baseline/sim_output.replay index 8d53abbb..4a968134 100644 Binary files a/src/scheduler/testsuite/fixtures/repl_trace_baseline/sim_output.replay and b/src/scheduler/testsuite/fixtures/repl_trace_baseline/sim_output.replay differ diff --git a/src/scheduler/testsuite/meson.build b/src/scheduler/testsuite/meson.build index cb4f1c6e..5d78725c 100644 --- a/src/scheduler/testsuite/meson.build +++ b/src/scheduler/testsuite/meson.build @@ -1634,6 +1634,23 @@ sched_limit_plugin_test = executable('sched_limit_plugin_test', dependencies: [scheduler_test_deps], install: false) +# external-license-resources Phase 3 (LSF 策略层): white-box regression +# coverage for the "GLOBAL limit on a GLOBAL|SHARED|EXTERNAL resource" +# shape (the per-project/per-queue/per-user license-quota mapping onto the +# existing limit vres mechanism). It #includes ../src/plugins/limit.c + +# limit_config.c directly (same convention as sched_limit_plugin_test +# above) so it drives the real match_job_global_limit_() -- `static`, +# otherwise unreachable. Unlike test_limit_plugin.c, its get_resvals()/ +# put_resvals() are FAITHFUL COPIES of resource.c's (the value path is the +# subject), and the fixture carries a real core->resmgr + +# sharedInst/sharedInUse -- see the file's own header comment. +sched_limit_project_scope_test = executable( + 'sched_limit_project_scope_test', + ['test_limit_project_scope.c'], + include_directories:[incdir], + dependencies: [scheduler_test_deps], + install: false) + # White-box regression test for bug-1686 (live SIGSEGV in # fcfs_cursor_enter_queue() <- builtin_fcfs_next_job(), a heap-use-after-free # on priv->cursor/job_cursor -- raw ordq_item_t* values that persist across @@ -2070,6 +2087,8 @@ test('scheduler snapshot load allocator regression test', env: ['ASAN_OPTIONS=detect_leaks=0']) test('scheduler limit plugin test', sched_limit_plugin_test, suite: 'scheduler', env: ['ASAN_OPTIONS=detect_leaks=0']) +test('scheduler limit project scope test', sched_limit_project_scope_test, + suite: 'scheduler', env: ['ASAN_OPTIONS=detect_leaks=0']) test('scheduler builtin fcfs cursor uaf test', sched_builtin_fcfs_cursor_uaf_test, suite: 'scheduler', env: ['ASAN_OPTIONS=detect_leaks=0']) diff --git a/src/scheduler/testsuite/test.c b/src/scheduler/testsuite/test.c index 2bf943b9..9e83f29c 100644 --- a/src/scheduler/testsuite/test.c +++ b/src/scheduler/testsuite/test.c @@ -730,6 +730,97 @@ test_core_safe_allocation_chain_driver_parity(void) destroy(core_sim->resmgr); } +/* + * external-license-resources Phase 1 (E2), design.md 3.3: the EXTERNAL + * accounting rule. An EXTERNAL resource's "total" as pushed by the extres + * daemon is already a FREE value F (license server's free count, net of + * both external consumers and our own checked-out jobs) -- so re-subtracting + * the scheduler's own usage bookkeeping (sharedInUse / w->resInUse) inside + * the admission formula would double-count X. The unsatisfy_ loop must + * therefore skip the inuse term for resources flagged XCE_RESF_EXTERNAL: + * + * EXTERNAL: total >= uncomit + wanted + * non-EXTERNAL: total >= inuse + uncomit + wanted (unchanged) + * + * Shared leg (sched_check_shared_satisfy_): a SHARED|EXTERNAL resource with + * total=F=4, sharedInUse=3, uncomit=0 must ADMIT wanted=4 (F >= 0+4) even + * though the legacy formula (4 < 3+0+4) would deny it -- and the same shape + * WITHOUT the EXTERNAL flag must still deny (legacy semantics pinned). + */ +static void test_unsatisfy_external_skips_inuse_shared_leg(void) +{ + xce_sched_core_t core; + xce_resource_t wire_resource; + xce_resource_t* made; + xce_resource_t* r = NULL; + xvec_t uncomitted; + xvec_idval_t wanted, unmeet; + xce_val_t total, inuse; + int rc; + + memset(&core, 0, sizeof(core)); + /* makeResource() -> resobj_init() -> makeString() needs the intern table + * (same per-test init as test_meet_host_selection_string_match). */ + xstring_table_init(139); + init_test_resmgr(&core); + zarray_init(&core.sharedInst, 4, NULL); + zarray_init(&core.sharedInUse, 4, NULL); + + /* Resource "lic": SHARED (cluster-wide pool) + EXTERNAL (free-value + * reporting) -- the shape mapping.py's `location: [external]` produces. + * idx 0. */ + memset(&wire_resource, 0, sizeof(wire_resource)); + wire_resource.name = "lic"; + wire_resource.idx = 0; + wire_resource.flags = XCE_RESF_SHARED | XCE_RESF_EXTERNAL; + made = (xce_resource_t*)makeResource(core.resmgr, + &wire_resource); + TEST_ASSERT_NOT_NULL(made); + TEST_ASSERT_TRUE(key_find(core.resmgr, (void*)"lic", (void**)&r) >= 0); + TEST_ASSERT_NOT_NULL(r); + + memset(&total, 0, sizeof(total)); + total.which = Value_u_fvalue; + total.u.fvalue = 4.0f; + memset(&inuse, 0, sizeof(inuse)); + inuse.which = Value_u_fvalue; + inuse.u.fvalue = 3.0f; + + zarray_put(&core.sharedInst, 0, &total); + zarray_put(&core.sharedInUse, 0, &inuse); + + xvec_init(&uncomitted, 1, sizeof(float)); + xvec_idval_init(&wanted, 1); + xvec_idval_store(&wanted, 0, 4.0f); + xvec_idval_init(&unmeet, 1); + + /* EXTERNAL: F=4 >= uncomit(0) + wanted(4) -> grant, despite inuse=3. */ + rc = sched_check_shared_satisfy(&core, &uncomitted, &wanted, &unmeet); + TEST_ASSERT_EQUAL_INT(1, rc); + + xvec_idval_fini(&wanted); + xvec_idval_fini(&unmeet); + + /* Same shape without EXTERNAL (plain SHARED pool): the legacy formula + * must still apply -- 4 < 3+0+4 -> deny. Overwrite the flag in place. */ + r->flags = XCE_RESF_SHARED; + + xvec_init(&uncomitted, 1, sizeof(float)); + xvec_idval_init(&wanted, 1); + xvec_idval_store(&wanted, 0, 4.0f); + xvec_idval_init(&unmeet, 1); + + rc = sched_check_shared_satisfy(&core, &uncomitted, &wanted, &unmeet); + TEST_ASSERT_EQUAL_INT(0, rc); + + xvec_idval_fini(&wanted); + xvec_idval_fini(&unmeet); + xvec_fini(&uncomitted); + zarray_fini(&core.sharedInst); + zarray_fini(&core.sharedInUse); + destroy(core.resmgr); +} + #define TEST(x) \ do { \ if (check_test(&arguments, #x)) { \ @@ -759,6 +850,7 @@ int main(int argc, char* argv[]) TEST(test_sched_update_existing_queue_null_payload); TEST(test_dead_pushmsg_symbols_absent); TEST(test_core_safe_allocation_chain_driver_parity); + TEST(test_unsatisfy_external_skips_inuse_shared_leg); return UNITY_END(); } diff --git a/src/scheduler/testsuite/test_core_events.c b/src/scheduler/testsuite/test_core_events.c index 7bf9f7ed..bb89310e 100644 --- a/src/scheduler/testsuite/test_core_events.c +++ b/src/scheduler/testsuite/test_core_events.c @@ -900,6 +900,99 @@ static void test_job_update_run_adopts_controller_allocation(void) "ownership of the allocation moves to the scheduler's job"); } +/* bug-1976 companion (external-license-resources Phase 3): the controller's + * RUN adoption is the scheduler's ONLY view of a dispatch (the controller is + * ground truth), so the adopted wire alloc must carry the shared-pool + * records too -- the limit vres charge among them (allocs_shared; the + * controller-side drop was bug-1976 itself, fixed in dispatch_a_job_()). + * Without them make_job_account() builds accounts_shared == NULL and + * sched_account_jobs() never charges the vres: sharedInUse stays 0.0 and + * unsatisfy_'s non-EXTERNAL branch (total >= inuse + uncomit + wanted) + * admits past every global limit forever. This pins the scheduler side of + * the round trip: an adoption whose wire alloc HAS shared records must + * produce accounts whose accounts_shared mirror them. */ +static void test_job_update_run_adoption_carries_shared_allocs(void) +{ + xce_sched_core_t core; + xce_core_event_t event; + xce_job_t* job1; + xce_job_t updated; + xce_sched_job_t* job; + xce_alloc_t* a_slots; + xce_host_alloc_t* h0; + xce_host_alloc_t* hallocs[1]; + xce_job_alloc_t* jalloc; + xce_alloc_t** shared; + int rc; + + fixture_init(&core, &g_full_plugin); + + job1 = make_job(71, JOB_STAT_PEND, "normal", "alice", "projA"); + + memset(&event, 0, sizeof(event)); + event.type = XCE_CORE_EVENT_JOBS_PUSH; + event.u.jobs_push.njobs = 1; + event.u.jobs_push.jobs = &job1; + rc = xce_core_push_events(&core, &event, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + job = sched_get_job(&core, 71); + TEST_ASSERT_NOT_NULL(job); + + /* the exact shape the controller re-sends post-fix: per-host slots + + * two shared records -- the limit vres instance charge and the + * underlying EXTERNAL pool draw. */ + a_slots = make_alloc((char*)"slots", NULL, 0, 1.0f, 0); + h0 = make_host_alloc((char*)"hostC", 1, make_allocs(1, a_slots)); + hallocs[0] = h0; + jalloc = make_job_alloc(71, 1, make_host_allocs(1, hallocs[0])); + + shared = make_allocs(2, + make_alloc((char*)"limit=extlic_share/p=projA", NULL, 0, 1.0f, 0), + make_alloc((char*)"extlic", NULL, 0, 1.0f, 0)); + TEST_ASSERT_NOT_NULL(shared); + jalloc->allocs_shared = shared; + jalloc->nres_shared = 2; + + memset(&updated, 0, sizeof(updated)); + updated.jobid = 71; + updated.state = JOB_STAT_RUN; + updated.allocs = jalloc; + + memset(&event, 0, sizeof(event)); + event.type = XCE_CORE_EVENT_JOB_UPDATE; + event.u.job_update.jobid = 71; + event.u.job_update.job = &updated; + rc = xce_core_push_events(&core, &event, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + TEST_ASSERT_EQUAL_UINT32(JOB_STAT_RUN, job->state); + TEST_ASSERT_NOT_NULL(job->allocs); + + /* the accounts the charge loop walks must mirror the shared records -- + * this is the exact property bug-1976's controller drop destroyed */ + TEST_ASSERT_NOT_NULL(job->accounts); + TEST_ASSERT_EQUAL_INT_MESSAGE(2, job->accounts->nres_shared, + "adoption must rebuild accounts WITH the shared records -- without " + "them sched_account_jobs() never charges the vres and every global " + "limit admits forever (bug-1976)"); + TEST_ASSERT_NOT_NULL(job->accounts->accounts_shared); + TEST_ASSERT_EQUAL_STRING("limit=extlic_share/p=projA", + job->accounts->accounts_shared[0]->resname); + TEST_ASSERT_EQUAL_FLOAT(1.0f, + job->accounts->accounts_shared[0]->reserved); + TEST_ASSERT_EQUAL_STRING("extlic", + job->accounts->accounts_shared[1]->resname); + TEST_ASSERT_EQUAL_FLOAT(1.0f, + job->accounts->accounts_shared[1]->reserved); + + /* allocation ownership moves; the job's own records survive intact */ + TEST_ASSERT_NULL(updated.allocs); + TEST_ASSERT_EQUAL_INT(2, job->allocs->nres_shared); + TEST_ASSERT_EQUAL_STRING("limit=extlic_share/p=projA", + job->allocs->allocs_shared[0]->resname); +} + /* Companion: a resync claiming a genuinely TERMINAL state (DONE/EXIT) for a * RUN-with-real-allocation job must still apply -- this is the legitimate * "job finished while this scheduler was disconnected" case bug-1734's own @@ -2005,9 +2098,140 @@ test_jobs_push_picker_rejection_still_tracks_job_and_succeeds(void) g_counts.job_push_rc = 0; } +/* external-license-resources Phase 1 (E2), landing-shape pin: a + * RESVALS_DELTA entry whose host is NULL or "" is a SHARED (cluster-level) + * value pushed by the extres daemon, not a per-worker value -- it must land + * in core->sharedInst at the resource's idx, mirroring snapshot_load()'s + * host==NULL/"" precedent (bug-1923) and scheduler.c's legacy DELTA loop + * needs the same rule. Controller's HOOK_DETAIL_RESOURCE_UPDATE notify is + * delivered to the core as RESVALS_DELTA (handler.c + * sched_handle_sched_update_resvals -> XCE_SCHED_OP_UPDATE_RESVALS), and + * worker pushes carry a NAMED host while the daemon's shared values carry + * host=="" -- so the empty host is the only marker the DELTA path has to + * route by. */ +static void test_resvals_delta_empty_host_lands_in_sharedinst(void) +{ + xce_sched_core_t core; + xce_core_event_t event; + xce_resource_t wire_resource; + xce_resource_t* resource_ptrs[1]; + xce_resval_t empty_host_resval; + xce_resval_t* delta_vals[1]; + xce_resval_list_t deltas; + xce_val_t val_empty; + xce_val_t* stored; + xce_resource_t* found = NULL; + int rc; + + fixture_init(&core, &g_full_plugin); + + memset(&wire_resource, 0, sizeof(wire_resource)); + wire_resource.name = "lic"; + wire_resource.idx = 3; + resource_ptrs[0] = &wire_resource; + + memset(&event, 0, sizeof(event)); + event.type = XCE_CORE_EVENT_RESOURCES_PUSH; + event.u.resources_push.n = 1; + event.u.resources_push.resources = resource_ptrs; + rc = xce_core_push_events(&core, &event, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&val_empty, 0, sizeof(val_empty)); + val_empty.which = Value_u_fvalue; + val_empty.u.fvalue = 5.0; + memset(&empty_host_resval, 0, sizeof(empty_host_resval)); + empty_host_resval.name = "lic"; + empty_host_resval.host = ""; + empty_host_resval.val = &val_empty; + + delta_vals[0] = &empty_host_resval; + deltas.count = 1; + deltas.vals = delta_vals; + + memset(&event, 0, sizeof(event)); + event.type = XCE_CORE_EVENT_RESVALS_DELTA; + event.u.resvals_delta.deltas = &deltas; + + rc = xce_core_push_events(&core, &event, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + TEST_ASSERT_TRUE(key_find(core.resmgr, (void*)"lic", + (void**)&found)); + TEST_ASSERT_NOT_NULL(found); + + stored = (xce_val_t*)zarray_get(&(core.sharedInst), found->idx); + TEST_ASSERT_NOT_NULL(stored); + TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which); + TEST_ASSERT_EQUAL_FLOAT(5.0, stored->u.fvalue); +} + +/* Same landing rule for the NULL spelling: snapshot_load (bug-1923) treats + * host==NULL as shared; the DELTA path must survive a NULL host (skip the + * per-worker lookup instead of strlen(NULL)) and land the value in + * core->sharedInst identically. */ +static void test_resvals_delta_null_host_lands_in_sharedinst(void) +{ + xce_sched_core_t core; + xce_core_event_t event; + xce_resource_t wire_resource; + xce_resource_t* resource_ptrs[1]; + xce_resval_t null_host_resval; + xce_resval_t* delta_vals[1]; + xce_resval_list_t deltas; + xce_val_t val_null; + xce_val_t* stored; + xce_resource_t* found = NULL; + int rc; + + fixture_init(&core, &g_full_plugin); + + memset(&wire_resource, 0, sizeof(wire_resource)); + wire_resource.name = "lic"; + wire_resource.idx = 3; + resource_ptrs[0] = &wire_resource; + + memset(&event, 0, sizeof(event)); + event.type = XCE_CORE_EVENT_RESOURCES_PUSH; + event.u.resources_push.n = 1; + event.u.resources_push.resources = resource_ptrs; + rc = xce_core_push_events(&core, &event, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + memset(&val_null, 0, sizeof(val_null)); + val_null.which = Value_u_fvalue; + val_null.u.fvalue = 4.0; + memset(&null_host_resval, 0, sizeof(null_host_resval)); + null_host_resval.name = "lic"; + null_host_resval.host = NULL; + null_host_resval.val = &val_null; + + delta_vals[0] = &null_host_resval; + deltas.count = 1; + deltas.vals = delta_vals; + + memset(&event, 0, sizeof(event)); + event.type = XCE_CORE_EVENT_RESVALS_DELTA; + event.u.resvals_delta.deltas = &deltas; + + rc = xce_core_push_events(&core, &event, 1); + TEST_ASSERT_EQUAL_INT(0, rc); + + TEST_ASSERT_TRUE(key_find(core.resmgr, (void*)"lic", + (void**)&found)); + TEST_ASSERT_NOT_NULL(found); + + stored = (xce_val_t*)zarray_get(&(core.sharedInst), found->idx); + TEST_ASSERT_NOT_NULL(stored); + TEST_ASSERT_EQUAL_INT(Value_u_fvalue, stored->which); + TEST_ASSERT_EQUAL_FLOAT(4.0, stored->u.fvalue); +} + int main(void) { UNITY_BEGIN(); + RUN_TEST(test_resvals_delta_empty_host_lands_in_sharedinst); + RUN_TEST(test_resvals_delta_null_host_lands_in_sharedinst); RUN_TEST(test_push_events_fires_each_hook_once_per_event_with_picker_idle); RUN_TEST(test_jobs_push_fires_on_job_push_once_per_job); RUN_TEST(test_queue_update_replaces_existing_interned_resreq); @@ -2016,6 +2240,7 @@ int main(void) RUN_TEST(test_jobs_push_duplicate_jobid_run_state_transfers_allocation); RUN_TEST(test_jobs_push_duplicate_jobid_controller_pend_wins_over_local_run); RUN_TEST(test_job_update_run_adopts_controller_allocation); + RUN_TEST(test_job_update_run_adoption_carries_shared_allocs); RUN_TEST(test_jobs_push_duplicate_jobid_terminal_resync_releases_real_allocation); RUN_TEST(test_jobs_push_duplicate_jobid_run_state_without_allocation_is_refused); RUN_TEST(test_jobs_push_duplicate_jobid_does_not_regress_terminal_state); diff --git a/src/scheduler/testsuite/test_limit_project_scope.c b/src/scheduler/testsuite/test_limit_project_scope.c new file mode 100644 index 00000000..35393f66 --- /dev/null +++ b/src/scheduler/testsuite/test_limit_project_scope.c @@ -0,0 +1,611 @@ +/* + * test_limit_project_scope.c - external-license-resources Phase 3 (LSF + * 策略层, design.md 3.5): per-project/per-queue/per-user token distribution + * maps onto the EXISTING limit vres mechanism. A limit on a GLOBAL + * underlying resource (e.g. the extlic license pool, SHARED|GLOBAL| + * EXTERNAL per mapping.py's location: [external] branch) with a projects: + * scope must: + * + * 1. match jobs by their obj->project string (literal project members -- + * make_g_member()'s literal fallback; named project groups are + * project-config's territory, deliberately not exercised here); + * 2. intern a usage instance named "limit=" (no /p= -- per_project + * false means one shared pool for all members of the project scope), + * or "limit=/p=" when per_project is set (independent + * usage per project under one configured cap); + * 3. write its cap into core->sharedInst at the vres's own idx + * (update_limit_res_value()); + * 4. take the matched vres into obj->glimits (match_job_global_limit_ + * -> xlist_store), so sched_fill_limit_requests() (resource.c) can + * stage the job's want into alloc_req.shared and the generic satisfy + * check can compare want+uncomit against the cap like any resource. + * + * The mechanical shape match_job_global_limit_ needs from the core: + * - core->resmgr: must already contain the UNDERLYING resource named by + * limit->val.resname (instantiate_limit() returns -ENOENT otherwise) + * AND grows the limit's vres via add_vres() -- exactly what + * sched_init_resources() does live; + * - core->sharedInst: the cap's zarray (update_limit_res_value()'s + * put_resvals target); + * - core->sharedInUse: where the generic check reads RUN jobs' usage + * (init here for parity; not touched by the plugin). + * + * White-box: #include's the real ../src/plugins/limit.c + + * limit_config.c (same convention as test_limit_plugin.c, whose fixture + * this file mirrors -- real group managers, real plugin private state). + * match_job_global_limit_() is `static`, otherwise unreachable outside + * its TU. get_resvals()/put_resvals() are NOT no-op stubs here (unlike + * test_limit_plugin.c, whose tests never exercise the value path): the + * value path IS this file's subject, so both are faithful copies of + * resource.c's (that file is not part of this TU; the copies preserve the + * production zarray protocol -- xce_val_t entries in Value_u_fvalue form, + * created zero-filled on first put, read as 0.0 when absent) so the + * fixture's sharedInst ends in the same state production would leave, + * and the cap assertions below prove the plugin put the RIGHT value at + * the RIGHT idx, not merely that it called something. + */ +#include +#include +#include + +#include + +#include "group.h" +#include "kvmgr.h" +#include "scheduler.h" +#include "xce/types.h" +#include "xcelimit.h" +#include "xstring.h" + +/* Faithful copies of resource.c's get_resvals()/put_resvals() (see file + * header). The plugin always calls them with nres == 1, so the copies + * need no dedup-by-duplicate-idx handling (resource.c's bug-1569 block is + * dead weight at nres 1). */ +int get_resvals(zarray_t* array, int nres, int* idxs, float* vals) +{ + xce_val_t* xvals[1]; + int rc; + + if ((array == NULL) || (nres < 0) || (idxs == NULL) || (vals == NULL)) { + return -EINVAL; + } + + if (nres == 0) { + return 0; + } + + memset(vals, 0x0, sizeof(*vals) * nres); + + rc = zarray_get_ex(array, nres, idxs, (void**)xvals); + if (rc < 0) { + return rc; + } + + if ((xvals[0] != NULL) && (xvals[0]->which == Value_u_fvalue)) { + vals[0] = xvals[0]->u.fvalue; + } + + return nres; +} + +int put_resvals(zarray_t* array, int nres, int* idxs, float* vals) +{ + xce_val_t* xvals[1]; + int rc; + + if ((array == NULL) || (nres < 0) || (idxs == NULL) || (vals == NULL)) { + return -EINVAL; + } + + if (nres == 0) { + return 0; + } + + rc = zarray_get_ex(array, nres, idxs, (void**)xvals); + if (rc < 0) { + return rc; + } + + if (idxs[0] < 0) { + return nres; + } + + if (xvals[0] == NULL) { + xce_val_t* val = (xce_val_t*)calloc(1, sizeof(*val)); + if (val == NULL) { + return -ENOMEM; + } + + val->which = Value_u_fvalue; + val->u.fvalue = 0.0; + zarray_put(array, idxs[0], val); + xvals[0] = val; + } + + xvals[0]->u.fvalue = vals[0]; + + return nres; +} + +/* limit_job_election() calls reset_sched_obj() on the REAL schedobj; + * this TU drives match_job_global_limit_() directly, so a stub purely + * for the linker -- same as test_limit_plugin.c. */ +void reset_sched_obj(xce_sched_obj_t* obj) +{ + (void)obj; +} + +int plugin_register(void* sched, void* plugin) +{ + (void)sched; + (void)plugin; + return 0; +} + +#include "../src/plugins/limit.c" +#include "../src/plugins/limit_config.c" + +void setUp(void) { return; } +void tearDown(void) { return; } + +/* Same fixture as test_limit_plugin.c's fixture_init(): real group + * managers for all four consumer kinds (build_limit()'s build_consumer() + * resolves each consumer against its own manager -- the "all" group for + * any scope the config leaves NULL), plus a real limit plugin private + * state. plugin->core is the bug-1964 honest reference. Extended here + * with the two pieces production's xce_sched_core_init() builds and the + * plugin's value path touches: core->resmgr (a kvmgr with no datakiller + * -- suite convention, detect_leaks=0, no teardown) and the + * sharedInst/sharedInUse zarays. */ +static void fixture_init(xce_scheduler_t* sched, xce_sched_plugin_t* plugin) +{ + kvmgr_params_t resmgr_params = { .key_type = KVMGR_KEY_STRING, + .nslots = 8, + .delta = 4, + .datakiller = NULL }; + int rc; + + memset(sched, 0, sizeof(*sched)); + + sched->core.resmgr = makeInstance(&xkvmgr_type, &resmgr_params); + TEST_ASSERT_NOT_NULL(sched->core.resmgr); + + groupmgr_init(&(sched->core.ugmgr), 4, 4); + groupmgr_init(&(sched->core.qgmgr), 4, 4); + groupmgr_init(&(sched->core.pgmgr), 4, 4); + groupmgr_init(&(sched->core.hgmgr), 4, 4); + TEST_ASSERT_NOT_NULL(sched->core.ugmgr); + TEST_ASSERT_NOT_NULL(sched->core.qgmgr); + TEST_ASSERT_NOT_NULL(sched->core.pgmgr); + TEST_ASSERT_NOT_NULL(sched->core.hgmgr); + + zarray_init(&(sched->core.sharedInst), 8, NULL); + zarray_init(&(sched->core.sharedInUse), 8, NULL); + + memset(plugin, 0, sizeof(*plugin)); + plugin->sched = sched; + plugin->core = &(sched->core); + + rc = limit_init(plugin); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_NOT_NULL(plugin->data); +} + +/* sched_init_resources()'s role, reduced to what the limit plugin needs: + * the UNDERLYING resource must exist in core->resmgr before the first + * match instantiates the vres (instantiate_limit()'s key_find of + * limit->val.resname returns -ENOENT otherwise -- the "unknown resource" + * failure mode test 3 pins). `flags` picks the underlying resource's own + * nature: SHARED|GLOBAL|EXTERNAL is the exact shape mapping.py's + * location: [external] branch produces for extlic. NOTE: makeResource_ + * assigns the resource's idx itself (the kvmgr slot), so the caller's + * wire.idx is ignored -- tests must read idx back, never assume it. */ +static xce_resource_t* register_resource(xce_sched_core_t* core, + const char* name, int flags) +{ + xce_resource_t wire; + xce_resource_t* made = NULL; + + memset(&wire, 0, sizeof(wire)); + wire.name = (char*)name; + wire.flags = flags; + + made = (xce_resource_t*)makeResource(core->resmgr, &wire); + TEST_ASSERT_NOT_NULL(made); + + return made; +} + +/* push limit configs through the plugin's config hook (both drivers do + * this -- plugin_config_update() live, plugin_config_update_core() sim) */ +static int push_limits(xce_sched_plugin_t* plugin, xce_limit_config_t** limits, + int n) +{ + xce_conf4sched_t config; + + memset(&config, 0, sizeof(config)); + config.nlimits = n; + config.limits = limits; + + return limit_update_config(plugin, &config); +} + +/* a heap limit config: scope = literal project members (heap group config + * -- the plugin's built limit references the group config's members for + * the consumer group's lifetime), absolute value cap */ +static xce_limit_config_t* make_project_limit(const char* name, + const char* resname, float cap, + const char* const* projects, + int nprojects) +{ + xce_limit_config_t* cfg = (xce_limit_config_t*)calloc(1, sizeof(*cfg)); + xce_limit_val_t* val = (xce_limit_val_t*)calloc(1, sizeof(*val)); + xce_group_config_t* g = (xce_group_config_t*)calloc(1, sizeof(*g)); + char** members = (char**)calloc((size_t)nprojects, + sizeof(char*)); + int i; + + TEST_ASSERT_NOT_NULL(cfg); + TEST_ASSERT_NOT_NULL(val); + TEST_ASSERT_NOT_NULL(g); + TEST_ASSERT_NOT_NULL(members); + + val->resname = (char*)resname; + val->percent = 0; + val->val = cap; + + for (i = 0; i < nprojects; i++) { + members[i] = (char*)projects[i]; + } + g->members.count = nprojects; + g->members.data = members; + + cfg->name = (char*)name; + cfg->val = val; + cfg->projects = g; + + return cfg; +} + +static void free_limit_config(xce_limit_config_t* cfg) +{ + FREEUP(cfg->val); + /* per_queue configs carry the scope in queues and NULL projects (the + * per_project shape is the mirror image) -- either group is optional */ + if (cfg->projects != NULL) { + FREEUP(cfg->projects->members.data); + FREEUP(cfg->projects); + } + if (cfg->queues != NULL) { + FREEUP(cfg->queues->members.data); + FREEUP(cfg->queues); + } + FREEUP(cfg); +} + +static xce_limit_t* find_limit(xce_sched_plugin_t* plugin, const char* name) +{ + limit_plugin_data_t* privdata = (limit_plugin_data_t*)plugin->data; + xce_limit_t* limit = NULL; + + key_find(privdata->limits, (void*)name, (void**)&limit); + + return limit; +} + +static xce_vres_t* find_instance(xce_sched_plugin_t* plugin, const char* name) +{ + limit_plugin_data_t* privdata = (limit_plugin_data_t*)plugin->data; + xce_vres_t* vres = NULL; + + key_find(privdata->instances, (void*)name, (void**)&vres); + + return vres; +} + +/* a minimal xce_sched_obj_t: project identity + the glimits xlist the + * plugin stores matched vres into. make_sched_obj() (schedobj.c) builds + * the real one; its project normalization ("default" for empty) is + * core_events.c's job, and the plugin consumes the result as-is -- so the + * fixture hands obj->project the string the scheduler would have. */ +static void make_obj(xce_sched_obj_t* obj, const char* project) +{ + memset(obj, 0, sizeof(*obj)); + obj->user = makeString("user1"); + obj->queue = makeString("normal"); + obj->project = makeString((char*)project); + xlist_init(&(obj->glimits), 8); +} + +/* drive match_job_global_limit_() the way limit_job_election() does: + * throwaway ctx, only ctx.schedobj is read. */ +static int match_global(xce_sched_plugin_t* plugin, xce_sched_obj_t* obj, + xce_limit_t* limit) +{ + xce_sched_ctx_t ctx = { 0 }; + + ctx.schedobj = obj; + + return match_job_global_limit_(plugin, &ctx, limit); +} + +/* ---------------------------------------------------------------------- */ +/* 1. A project-scoped limit on a GLOBAL|SHARED|EXTERNAL resource matches + * by obj->project (literal member), is global (lands in glimits -- + * is_host_based_limit() keys on the HOST consumer, never the underlying + * resource's own GLOBAL flag), and instantiates the usage instance named + * "limit=" (per_project false -> one shared pool for the project + * scope; no /p= segment). Members outside the scope match nothing and + * store nothing. */ +static void test_project_scope_match_and_instantiate(void) +{ + xce_scheduler_t sched; + xce_sched_plugin_t plugin; + xce_sched_obj_t obj; + xce_limit_config_t* limits[1]; + xce_limit_t* limit; + xce_vres_t* vres; + const char* const scope_a[] = { "projA" }; + char vresname[64]; + int rc; + + fixture_init(&sched, &plugin); + + /* the extlic shape: SHARED (cluster pool) + GLOBAL (admitted in the + * shared leg) + EXTERNAL (free-value reporting; Phase 1's unsatisfy_ + * rule -- asserted independently in testsuite/test.c). */ + register_resource(&(sched.core), "extlic", + XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL); + + limits[0] = make_project_limit("extlic_projA", "extlic", 2.0f, scope_a, 1); + TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 1)); + + limit = find_limit(&plugin, "extlic_projA"); + TEST_ASSERT_NOT_NULL(limit); + + /* in-scope project matches. match_job_global_limit_()'s own rc + * contract is "0 on success" (its tail rc = xlist_store() returns the + * stored idx -- 0 for the first entry; only < 0 is failure, the same + * rc<0 break limit_job_election() applies), so the success signal is + * rc >= 0 AND the vres landing in obj.glimits -- asserted below. */ + make_obj(&obj, "projA"); + rc = match_global(&plugin, &obj, limit); + TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc); + + /* the instance is keyed by the bare vres name (per_project false) */ + snprintf(vresname, sizeof(vresname), "limit=extlic_projA"); + vres = find_instance(&plugin, vresname); + TEST_ASSERT_NOT_NULL(vres); + TEST_ASSERT_EQUAL_PTR(limit, vres->limit); + TEST_ASSERT_NOT_NULL(vres->vres); + TEST_ASSERT_EQUAL_STRING(vresname, vres->vres->name); + /* the vres is a REAL resource now: SHARED|GLOBAL (its own nature, from + * add_vres()'s `shared` flag derived from is_host_based_limit()) + + * LIMIT (the satisfy check's PREASON_LIMIT_GLOBAL discriminator). */ + TEST_ASSERT_TRUE((vres->vres->flags & XCE_RESF_LIMIT) != 0); + TEST_ASSERT_TRUE((vres->vres->flags & XCE_RESF_SHARED) != 0); + TEST_ASSERT_TRUE((vres->vres->flags & XCE_RESF_GLOBAL) != 0); + /* the underlying resource is linked */ + TEST_ASSERT_NOT_NULL(vres->res); + TEST_ASSERT_EQUAL_STRING("extlic", vres->res->name); + + /* the matched vres was stored into obj->glimits for + * sched_fill_limit_requests() to stage */ + TEST_ASSERT_EQUAL_INT(1, obj.glimits.count); + TEST_ASSERT_EQUAL_PTR(vres, xlist_get(&(obj.glimits), 0)); + + /* out-of-scope project: no match, nothing stored */ + make_obj(&obj, "projB"); + rc = match_global(&plugin, &obj, limit); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(0, obj.glimits.count); + + free_limit_config(limits[0]); +} + +/* ---------------------------------------------------------------------- */ +/* 2. The cap lands in core->sharedInst at the vres's own idx, and the + * per_project variant slices by project ("limit=/p="): two + * projects get independent usage entries under one configured cap. */ +static void test_cap_value_and_per_project_instance_names(void) +{ + xce_scheduler_t sched; + xce_sched_plugin_t plugin; + xce_sched_obj_t obj; + xce_limit_config_t* limits[2]; + xce_limit_t* shared_pool; + xce_limit_t* per_project; + xce_vres_t* vres; + xce_val_t* v; + const char* const scope_a[] = { "projA" }; + const char* const scope_ab[] = { "projA", "projB" }; + char name[64]; + int rc; + + fixture_init(&sched, &plugin); + + register_resource(&(sched.core), "extlic", + XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL); + + limits[0] = make_project_limit("extlic_pool", "extlic", 2.0f, scope_a, 1); + /* per-project slicing needs BOTH projects in the limit's scope: the + * scope decides who matches at all; per_project then splits each + * match into its own "limit=/p=" usage instance. */ + limits[1] = make_project_limit("extlic_share", "extlic", 3.0f, scope_ab, + 2); + limits[1]->per_project = 1; + TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 2)); + + /* shared-pool limit: cap written at the instance's own vres idx */ + shared_pool = find_limit(&plugin, "extlic_pool"); + TEST_ASSERT_NOT_NULL(shared_pool); + make_obj(&obj, "projA"); + TEST_ASSERT_GREATER_OR_EQUAL_INT(0, match_global(&plugin, &obj, + shared_pool)); + + vres = (xce_vres_t*)xlist_get(&(obj.glimits), 0); + TEST_ASSERT_NOT_NULL(vres); + v = (xce_val_t*)zarray_get(&(sched.core.sharedInst), vres->vres->idx); + TEST_ASSERT_NOT_NULL(v); + TEST_ASSERT_EQUAL_FLOAT(2.0f, v->u.fvalue); + + /* per-project limit: /p= slices the name, so projA and projB get + * independent instances (independent usage) under one cap */ + per_project = find_limit(&plugin, "extlic_share"); + TEST_ASSERT_NOT_NULL(per_project); + + make_obj(&obj, "projA"); + rc = match_global(&plugin, &obj, per_project); + TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc); + make_obj(&obj, "projB"); + rc = match_global(&plugin, &obj, per_project); + TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc); + + snprintf(name, sizeof(name), "limit=extlic_share/p=projA"); + TEST_ASSERT_NOT_NULL(find_instance(&plugin, name)); + snprintf(name, sizeof(name), "limit=extlic_share/p=projB"); + TEST_ASSERT_NOT_NULL(find_instance(&plugin, name)); + + free_limit_config(limits[0]); + free_limit_config(limits[1]); +} + +/* ---------------------------------------------------------------------- */ +/* 2b. The per-queue distribution shape (limits.yaml's extlic_qdist): a + * queues:-scoped limit with per_queue slices its usage instance by queue + * ("limit=/q=") exactly as the projects shape slices by + * project ("/p=") -- how a site partitions one license pool across + * queues. Also pins that the queue scope does NOT match a project-only + * identity: the consumer matched is the obj's queue, so a different + * queue outside the members list matches nothing. */ +static void test_queue_scope_per_queue_instance_names(void) +{ + xce_scheduler_t sched; + xce_sched_plugin_t plugin; + xce_sched_obj_t obj; + xce_limit_config_t* limits[1]; + xce_limit_config_t* cfg; + xce_group_config_t* qg; + xce_vres_t* vres; + const char* const scope_a[] = { "projA" }; + const char* const queues[] = { "normal", "night" }; + char** qmembers; + char name[64]; + int rc; + int i; + + fixture_init(&sched, &plugin); + + register_resource(&(sched.core), "extlic", + XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL); + + limits[0] = make_project_limit("extlic_qdist", "extlic", 4.0f, scope_a, + 1); + cfg = limits[0]; + FREEUP(cfg->projects->members.data); + FREEUP(cfg->projects); + cfg->projects = NULL; + cfg->per_queue = 1; + + /* queue scope: heap group config, same lifetime contract as the + * project one make_project_limit() builds */ + qg = (xce_group_config_t*)calloc(1, sizeof(*qg)); + qmembers = (char**)calloc(2, sizeof(char*)); + TEST_ASSERT_NOT_NULL(qg); + TEST_ASSERT_NOT_NULL(qmembers); + for (i = 0; i < 2; i++) { + qmembers[i] = (char*)queues[i]; + } + qg->members.count = 2; + qg->members.data = qmembers; + cfg->queues = qg; + + TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 1)); + + /* each in-scope queue gets its OWN /q= instance */ + make_obj(&obj, "projA"); + obj.queue = makeString("normal"); + rc = match_global(&plugin, &obj, find_limit(&plugin, "extlic_qdist")); + TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc); + + snprintf(name, sizeof(name), "limit=extlic_qdist/q=normal"); + vres = find_instance(&plugin, name); + TEST_ASSERT_NOT_NULL_MESSAGE(vres, + "per_queue must slice the usage instance by queue (/q= segment), " + "the queue analogue of the /p= project shape"); + TEST_ASSERT_EQUAL_INT(1, obj.glimits.count); + TEST_ASSERT_EQUAL_PTR(vres, xlist_get(&(obj.glimits), 0)); + + /* cap 4.0 written at this instance's own vres idx */ + { + xce_val_t* v = (xce_val_t*)zarray_get(&(sched.core.sharedInst), + vres->vres->idx); + TEST_ASSERT_NOT_NULL(v); + TEST_ASSERT_EQUAL_FLOAT(4.0f, v->u.fvalue); + } + + make_obj(&obj, "projA"); + obj.queue = makeString("night"); + rc = match_global(&plugin, &obj, find_limit(&plugin, "extlic_qdist")); + TEST_ASSERT_GREATER_OR_EQUAL_INT(0, rc); + snprintf(name, sizeof(name), "limit=extlic_qdist/q=night"); + TEST_ASSERT_NOT_NULL(find_instance(&plugin, name)); + + /* out-of-scope queue matches nothing */ + make_obj(&obj, "projA"); + obj.queue = makeString("short"); + rc = match_global(&plugin, &obj, find_limit(&plugin, "extlic_qdist")); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(0, obj.glimits.count); + + free_limit_config(limits[0]); +} + +/* ---------------------------------------------------------------------- */ +/* 3. The unknown-underlying-resource failure is a clean -ENOENT: a limit + * naming a resource the core never registered must not crash and must + * leave nothing in obj->glimits (the failure mode instantiate_limit()'s + * own comment warns of -- the resname must exist before the first match). */ +static void test_unknown_underlying_resource_fails_clean(void) +{ + xce_scheduler_t sched; + xce_sched_plugin_t plugin; + xce_sched_obj_t obj; + xce_limit_config_t* limits[1]; + const char* const scope_a[] = { "projA" }; + int rc; + + fixture_init(&sched, &plugin); + /* deliberately NO resource registered */ + + limits[0] = make_project_limit("ghost_limit", "ghost_lic", 2.0f, scope_a, + 1); + TEST_ASSERT_EQUAL_INT(0, push_limits(&plugin, limits, 1)); + TEST_ASSERT_NOT_NULL(find_limit(&plugin, "ghost_limit")); + + make_obj(&obj, "projA"); + rc = match_global(&plugin, &obj, find_limit(&plugin, "ghost_limit")); + TEST_ASSERT_EQUAL_INT(-ENOENT, rc); + TEST_ASSERT_EQUAL_INT(0, obj.glimits.count); + + free_limit_config(limits[0]); +} + +int main(void) +{ + int rc; + + /* group names are interned strings (makeString(), group.c) */ + xstring_table_init(139); + + UNITY_BEGIN(); + RUN_TEST(test_project_scope_match_and_instantiate); + RUN_TEST(test_cap_value_and_per_project_instance_names); + RUN_TEST(test_queue_scope_per_queue_instance_names); + RUN_TEST(test_unknown_underlying_resource_fails_clean); + rc = UNITY_END(); + + xstring_table_fini(); + + return rc; +} diff --git a/src/schema/controller.capnp b/src/schema/controller.capnp index 446562b7..879d9869 100644 --- a/src/schema/controller.capnp +++ b/src/schema/controller.capnp @@ -153,6 +153,11 @@ enum HostRole { scheduler @2; controller @3; apiserver @4; + # external-license-resources Phase 1: the external-resources daemon + # (xceesd) joins with this role to push shared-resource free counts + # (license totals etc.) through the resval update pipeline. Appended + # last so existing enum values keep their wire numbers. + extres @5; } struct JoinData $C.mapname("xce_joindata_t") { diff --git a/src/worker/include/elim.h b/src/worker/include/elim.h new file mode 100644 index 00000000..bf3e5aef --- /dev/null +++ b/src/worker/include/elim.h @@ -0,0 +1,83 @@ +#ifndef ELIM_H +#define ELIM_H + +#include +#include + +/* + * elim -- the LSF-elim-compatible external resource collector for xcecd + * (external-license-resources design §3.3b). Each configured source names a + * command run on this host (posix_spawn, watchdog kill at + * elim_script_timeout_ms); the command prints " " (elim + * format) or a bare number on stdout, and the value registers as a custom + * load index on resource_sync's registry -- flowing the live per-host + * report chain (collect_load_indices -> xcecd_send_load_indices -> + * controller update_val_res -> scheduler RESVALS_DELTA) with zero + * downstream changes. + * + * Per-host semantics by design: an elim script answers "how much does THIS + * host have"; cluster-global pools (license servers) belong to xceesd. + */ + +typedef struct elim_source_s { + char* name; /* resource name the script reports */ + char* command; /* shell command line (run via /bin/sh -c) */ + int interval; /* seconds between re-runs; 0 = every collect */ +} elim_source_t; + +typedef struct elim_config_s { + int nsrc; + elim_source_t* src; /* array of nsrc sources */ +} elim_config_t; + +/* + * Parse the elim section of the worker config file: + * + * external_resources: + * - name: dynres1 + * command: /opt/xce/scripts/report-dynres1.sh + * interval: 30 + * + * Returns 0, -ENOENT (file missing), -EINVAL (bad shape: a source without + * name or command can never report -- fail loudly at load, not silently at + * runtime), -ENOMEM. + */ +extern int elim_config_load(const char* path, elim_config_t* cfg_out); +extern int elim_config_load_path(const char* dir, const char* name, + elim_config_t* cfg_out); +extern void elim_config_fini(elim_config_t* cfg); + +/* + * Register every configured source as a custom (non-builtin) load index on + * resource_sync's registry. Returns 0, or -EEXIST if a source's name is + * already taken (another source, or one of the 11 builtins -- a user script + * must not shadow "mem"/"r15s"/...), or -EINVAL/-ENOMEM. Registration + * happens at startup, before the sync thread's first collect -- the + * registry self-initializes for that (the pre-phase "capacity reached (0)" + * trap). + */ +extern int elim_register_sources(const elim_config_t* cfg); +extern void elim_unregister_sources(const elim_config_t* cfg); + +/* + * Run one collection command and read its stdout. Returns 0 with `out` + * NUL-terminated, -ETIMEDOUT when the watchdog killed the child, + * -ENODATA on empty output, -EINVAL/-errno on spawn/read failure. + * The child gets an empty signal mask and default dispositions; stderr is + * /dev/null. Never blocks past elim_script_timeout_ms. + */ +extern int elim_script_collect(const char* command, char* out, size_t outsz); + +/* + * Parse an elim-format collection output: the first non-empty line carries + * " " or a bare number; leading/trailing whitespace ignored. + * Returns the value, or NAN on garbage/empty (caller owns the fail-open + * policy). + */ +extern double elim_parse_script_output(const char* out); + +/* Watchdog, milliseconds a collection command may run before it is killed + * (overridable from tests / future CLI). */ +extern int elim_script_timeout_ms; + +#endif // ELIM_H diff --git a/src/worker/include/resource_sync.h b/src/worker/include/resource_sync.h index e271b11c..f43e926d 100644 --- a/src/worker/include/resource_sync.h +++ b/src/worker/include/resource_sync.h @@ -19,4 +19,10 @@ bool resource_sync_register_load_index(const char* name, int valtype, bool is_builtin); bool resource_sync_unregister_load_index(const char* name); +/* Ensure the registry array exists and the 11 builtin load indices are in + * it -- for external registrars (elim) that run before the sync thread's + * first collect would lazily populate them, so their name-collision checks + * see the builtins. */ +void resource_sync_ensure_builtin_load_indices(void); + #endif // RESOURCE_SYNC_H \ No newline at end of file diff --git a/src/worker/include/xcecd.h b/src/worker/include/xcecd.h index 480fcba1..83d6d007 100644 --- a/src/worker/include/xcecd.h +++ b/src/worker/include/xcecd.h @@ -2,6 +2,7 @@ #define XCECD_H #include "controller_connector.h" +#include "elim.h" #include "hardware_monitor.h" #include "job_information.h" #include "node_information.h" @@ -63,6 +64,11 @@ typedef struct { * xcecd_start() already ran. candidates_count == 0 means no --peers was * given (single-host default). */ xce_cluster_config_t cluster_cfg; + /* elim sources from the -c config file's external_resources section + * (design §3.3b). Loaded in main.c (elim_config_load); ownership moves + * to xcecd_ctx by xcecd_init(), which registers the sources with + * resource_sync at startup. */ + elim_config_t elim; } xcecd_config_t; typedef struct { @@ -88,6 +94,9 @@ typedef struct { char* log_dir; char* db_path; char* virtual_name; + /* elim collectors (design §3.3b): sources loaded from the -c config + * file, registered with resource_sync by xcecd_init(). */ + elim_config_t elim; /* Copied from xcecd_config_t by xcecd_init(); consumed by * controller_connector_init() (xcecd_start()) to populate * cc_ctx->cluster_cfg BEFORE controller_connector_start() fires the diff --git a/src/worker/meson.build b/src/worker/meson.build index 4d97789b..4245a853 100644 --- a/src/worker/meson.build +++ b/src/worker/meson.build @@ -16,6 +16,7 @@ xcecd_worker_srcs = [ 'src/xcecd.c', 'src/connection_handler.c', 'src/resource_sync.c', + 'src/elim.c', 'src/node_information.c', 'src/thread_pool.c', 'src/hardware_monitor.c', @@ -95,6 +96,7 @@ xcecd_srcs = [ 'src/connection_handler.c', 'src/controller_connector.c', 'src/resource_sync.c', + 'src/elim.c', 'src/xcecd_worker_utils.c', 'src/xcecd_resync.c', 'src/hardware_monitor.c', diff --git a/src/worker/src/elim.c b/src/worker/src/elim.c new file mode 100644 index 00000000..b15400fb --- /dev/null +++ b/src/worker/src/elim.c @@ -0,0 +1,656 @@ +/* elim.c -- the LSF-elim-compatible external resource collector for xcecd + * (external-license-resources design §3.3b). + * + * The worker already owns half the elim model as live code: resource_sync.c + * keeps a load-index registry (11 builtins shaped after LSF's load index + * name set) whose values flow the per-host report chain every 15 ticks -- + * collect_load_indices() -> xcecd_send_load_indices() -> controller + * update_val_res (non-SHARED lands in w->host.resvals) -> scheduler + * RESVALS_DELTA -> sched_check_host_satisfy_. What was missing is the + * ENTRY: a mechanism for a user to register their own index. This module is + * that entry -- config-driven script collectors registered as custom + * (non-builtin) load indices at startup. + * + * posix_spawn(), not fork(): xcecd is multi-threaded (no-fork rule, + * .kiro/steering/tech.md "Process Creation") and a forked child that + * allocated or logged before exec could deadlock on a lock another thread + * held. The spawn/watchdog shape mirrors esd_script.c (the xceesd + * collector, same phase's global leg): pipe, poll(2) in 100ms slices, + * kill(SIGKILL) at elim_script_timeout_ms, waitpid reaps -- a hung script + * must not stall the sync thread. + */ +#define _GNU_SOURCE + +#include "elim.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "resource_sync.h" +#include "xcelog.h" + +/* Watchdog, milliseconds of wall clock a collection command may run before + * it is killed (design §3.3b: 脚本超时看门狗 kill). */ +int elim_script_timeout_ms = 10 * 1000; + +/* ── config parse (libcyaml, esd_config.c pattern) ────────────────────── */ + +typedef struct yaml_elim_source_s { + char* name; + char* command; + long interval; /* 0 = every collect tick */ +} yaml_elim_source_t; + +typedef struct yaml_elim_config_s { + yaml_elim_source_t* external_resources; + unsigned external_resources_count; +} yaml_elim_config_t; + +static const cyaml_schema_field_t elim_source_fields_schema[] = { + CYAML_FIELD_STRING_PTR("name", CYAML_FLAG_POINTER, yaml_elim_source_t, + name, 1, CYAML_UNLIMITED), + CYAML_FIELD_STRING_PTR("command", CYAML_FLAG_POINTER, yaml_elim_source_t, + command, 1, CYAML_UNLIMITED), + CYAML_FIELD_INT("interval", CYAML_FLAG_OPTIONAL, yaml_elim_source_t, + interval), + CYAML_FIELD_END +}; + +static const cyaml_schema_value_t elim_source_schema = { + CYAML_VALUE_MAPPING(CYAML_FLAG_DEFAULT, yaml_elim_source_t, + elim_source_fields_schema) +}; + +static const cyaml_schema_field_t elim_config_fields_schema[] = { + CYAML_FIELD_SEQUENCE("external_resources", CYAML_FLAG_POINTER, + yaml_elim_config_t, external_resources, + &elim_source_schema, 0, CYAML_UNLIMITED), + CYAML_FIELD_END +}; + +static const cyaml_schema_value_t elim_config_schema = { + CYAML_VALUE_MAPPING(CYAML_FLAG_POINTER, yaml_elim_config_t, + elim_config_fields_schema) +}; + +int elim_config_load_path(const char* dir, const char* name, + elim_config_t* cfg_out) +{ + static cyaml_config_t cyaml_cfg = { + .mem_fn = cyaml_mem, + .log_level = CYAML_LOG_WARNING, + .flags = CYAML_CFG_IGNORE_UNKNOWN_KEYS, + }; + + yaml_elim_config_t* ycfg = NULL; + cyaml_err_t err; + char* path = NULL; + elim_config_t cfg; + int i; + int rc; + + if ((dir == NULL) || (name == NULL) || (cfg_out == NULL)) { + return -EINVAL; + } + + memset(cfg_out, 0, sizeof(*cfg_out)); + memset(&cfg, 0, sizeof(cfg)); + + if (asprintf(&path, "%s/%s", dir, name) < 0) { + return -ENOMEM; + } + + err = cyaml_load_file(path, &cyaml_cfg, &elim_config_schema, + (void**)&ycfg, NULL); + free(path); + if (err != CYAML_OK) { + /* Map file-not-found apart from bad-file so the startup log can + * say which (esd_config.c's probe). */ + char probe[512]; + FILE* f; + + snprintf(probe, sizeof(probe), "%s/%s", dir, name); + f = fopen(probe, "r"); + if (f == NULL) { + return -ENOENT; + } + fclose(f); + return -EINVAL; + } + + if (ycfg == NULL) { + return -EINVAL; + } + + cfg.nsrc = (int)ycfg->external_resources_count; + cfg.src = NULL; + + if (cfg.nsrc > 0) { + cfg.src = (elim_source_t*)calloc((size_t)cfg.nsrc, + sizeof(elim_source_t)); + if (cfg.src == NULL) { + cyaml_free(&cyaml_cfg, &elim_config_schema, ycfg, 0); + return -ENOMEM; + } + } + + rc = 0; + for (i = 0; i < cfg.nsrc; i++) { + yaml_elim_source_t* ys = &(ycfg->external_resources[i]); + + /* A source without name or command can never report -- fail + * loudly at load, not silently at runtime. */ + if ((ys->name == NULL) || (ys->command == NULL)) { + rc = -EINVAL; + break; + } + + cfg.src[i].name = strdup(ys->name); + cfg.src[i].command = strdup(ys->command); + cfg.src[i].interval = (int)ys->interval; + + if ((cfg.src[i].name == NULL) || (cfg.src[i].command == NULL)) { + rc = -ENOMEM; + break; + } + } + + cyaml_free(&cyaml_cfg, &elim_config_schema, ycfg, 0); + + if (rc < 0) { + elim_config_fini(&cfg); + return rc; + } + + *cfg_out = cfg; + + return 0; +} + +int elim_config_load(const char* path, elim_config_t* cfg_out) +{ + char* dir; + char* name; + char* slash; + int rc; + + if ((path == NULL) || (cfg_out == NULL)) { + return -EINVAL; + } + + dir = strdup(path); + if (dir == NULL) { + return -ENOMEM; + } + + slash = strrchr(dir, '/'); + if (slash == NULL) { + name = dir; + dir = strdup("."); + if (dir == NULL) { + free(name); + return -ENOMEM; + } + } else { + *slash = '\0'; + name = slash + 1; + if (*name == '\0') { + free(dir); + return -EINVAL; + } + if (dir == name - 1) { + /* path was "/name" -- dir is "" */ + free(dir); + dir = strdup("/"); + if (dir == NULL) { + return -ENOMEM; + } + } + } + + rc = elim_config_load_path(dir, name, cfg_out); + + free(dir); + + return rc; +} + +void elim_config_fini(elim_config_t* cfg) +{ + int i; + + if (cfg == NULL) { + return; + } + + for (i = 0; i < cfg->nsrc; i++) { + free(cfg->src[i].name); + free(cfg->src[i].command); + } + free(cfg->src); + + memset(cfg, 0, sizeof(*cfg)); +} + +/* ── output parse (elim contract) ─────────────────────────────────────── */ + +/* + * The elim contract: the first non-empty line carries " " or + * a bare number; leading/trailing whitespace ignored; anything else is + * garbage -> NAN (the caller owns the fail-open policy). Same shape as + * esd_parse_script_output -- a number is a number wherever it stands. + */ +double elim_parse_script_output(const char* out) +{ + const char* p; + char* endp = NULL; + double v; + + if (out == NULL) { + return NAN; + } + + p = out; + while (*p != '\0') { + while ((*p == ' ') || (*p == '\t') || (*p == '\r') || (*p == '\n')) { + p++; + } + if (*p == '\0') { + break; + } + + v = strtod(p, &endp); + if ((endp != NULL) && (endp != p)) { + return v; + } + + /* skip this non-numeric token and try the next one */ + while ((*p != '\0') && (*p != ' ') && (*p != '\t') && (*p != '\r') + && (*p != '\n')) { + p++; + } + } + + return NAN; +} + +/* ── script runner (posix_spawn + watchdog, esd_script.c shape) ───────── */ + +static int elim_script_spawn(const char* command, int out_fd, pid_t* pid_out) +{ + posix_spawn_file_actions_t fa; + posix_spawnattr_t attr; + sigset_t none, all; + char* argv[] = { (char*)"sh", (char*)"-c", (char*)command, NULL }; + extern char** environ; + pid_t pid; + int rc; + + sigemptyset(&none); + sigfillset(&all); + + rc = posix_spawn_file_actions_init(&fa); + if (rc == 0) { + rc = posix_spawn_file_actions_adddup2(&fa, out_fd, STDOUT_FILENO); + } + if (rc == 0) { + rc = posix_spawn_file_actions_addopen(&fa, STDERR_FILENO, "/dev/null", + O_WRONLY, 0); + } + if (rc == 0) { + rc = posix_spawnattr_init(&attr); + } + if (rc == 0) { + rc = posix_spawnattr_setflags( + &attr, POSIX_SPAWN_SETSIGMASK | POSIX_SPAWN_SETSIGDEF); + } + if (rc == 0) { + rc = posix_spawnattr_setsigmask(&attr, &none); + } + if (rc == 0) { + rc = posix_spawnattr_setsigdefault(&attr, &all); + } + if (rc == 0) { + rc = posix_spawn(&pid, "/bin/sh", &fa, &attr, argv, environ); + } + posix_spawnattr_destroy(&attr); + posix_spawn_file_actions_destroy(&fa); + + if (rc != 0) { + return -rc; + } + + *pid_out = pid; + + return 0; +} + +int elim_script_collect(const char* command, char* out, size_t outsz) +{ + int fds[2]; + pid_t pid = -1; + size_t total = 0; + int elapsed_ms = 0; + int rc; + int wstatus = 0; + struct pollfd pfd; + + if ((command == NULL) || (*command == '\0') || (out == NULL) + || (outsz == 0)) { + return -EINVAL; + } + + out[0] = '\0'; + + if (pipe(fds) != 0) { + rc = -errno; + XCECD_LOG(XCE_LOG_ERROR, "elim: pipe failed (%s)", strerror(-rc)); + return rc; + } + + rc = elim_script_spawn(command, fds[1], &pid); + close(fds[1]); /* parent's copy -- child holds the write end now */ + if (rc < 0) { + close(fds[0]); + XCECD_LOG(XCE_LOG_ERROR, "elim: spawn failed for \"%s\" (%s)", command, + strerror(-rc)); + return rc; + } + + /* Bound the read at the watchdog deadline: a script that hangs must + * not stall the sync thread -- the value it was going to report + * belongs to a dead world anyway. */ + pfd.fd = fds[0]; + pfd.events = POLLIN; + + while (total < outsz - 1) { + int pret = poll(&pfd, 1, 100); + + if (pret < 0) { + if (errno == EINTR) { + continue; + } + rc = -errno; + XCECD_LOG(XCE_LOG_ERROR, "elim: poll failed for \"%s\" (%s)", + command, strerror(-rc)); + kill(pid, SIGKILL); + waitpid(pid, &wstatus, 0); + close(fds[0]); + return rc; + } + if (pret == 0) { + elapsed_ms += 100; + if (elapsed_ms >= elim_script_timeout_ms) { + XCECD_LOG(XCE_LOG_WARN, + "elim: command \"%s\" exceeded %d ms -- killing " + "it, value not refreshed this round", + command, elim_script_timeout_ms); + kill(pid, SIGKILL); + waitpid(pid, &wstatus, 0); + close(fds[0]); + return -ETIMEDOUT; + } + continue; + } + + if (pfd.revents & (POLLERR | POLLHUP)) { + ssize_t n = read(fds[0], out + total, outsz - 1 - total); + if (n > 0) { + total += (size_t)n; + } + break; + } + if (pfd.revents & POLLIN) { + ssize_t n = read(fds[0], out + total, outsz - 1 - total); + if (n <= 0) { + break; /* EOF or error: done reading */ + } + total += (size_t)n; + } + } + + out[total] = '\0'; + close(fds[0]); + + /* reap regardless of which path brought us here */ + while (waitpid(pid, &wstatus, 0) < 0) { + if (errno != EINTR) { + break; + } + } + + if (total == 0) { + XCECD_LOG(XCE_LOG_WARN, + "elim: command \"%s\" produced no output", command); + return -ENODATA; + } + + return 0; +} + +/* ── getter plumbing: per-source cache behind the context-free API ────── */ + +/* resource_sync's getter signature is context-free (char* (*)(void)) -- the + * per-source state (command, cached value, next-refresh deadline) lives in + * this static array, indexed by registration order. MAX_LOAD_INDICES (64) + * bounds the registry; the config loader's own parse bounds us to a sane + * subset of that, and elim_register_sources' capacity failure surfaces at + * startup. */ +typedef struct { + char* command; + int interval; /* seconds; 0 = refresh every collect */ + double value; /* last good parse */ + bool have_value; + time_t next_run; /* epoch after which the script re-runs */ +} elim_source_state_t; + +static elim_source_state_t* g_elim_states = NULL; +static int g_elim_nstates = 0; +static pthread_mutex_t g_elim_mutex = PTHREAD_MUTEX_INITIALIZER; + +static char* elim_getter_run(int slot) +{ + elim_source_state_t* st = &g_elim_states[slot]; + char buf[4096]; + int rc; + double v; + char* out_str; + time_t now; + + /* interval cache: 0 = run the script on every collect; otherwise the + * script re-runs only once its interval has elapsed since the last + * good run, and the cached value is served in between (the send + * cadence is the sync thread's 15-tick cycle regardless -- the interval + * governs script-run frequency, not push frequency). */ + now = time(NULL); + if (st->have_value && (st->interval > 0) && (now < st->next_run)) { + out_str = (char*)malloc(64); + if (out_str == NULL) { + return NULL; + } + snprintf(out_str, 64, "%f", st->value); + return out_str; + } + + rc = elim_script_collect(st->command, buf, sizeof(buf)); + if (rc < 0) { + /* Fail-open: keep the last known value (the collector died but the + * host is still there; the sync thread keeps reporting). */ + if (!st->have_value) { + return strdup("0"); + } + return strdup(""); + } + + v = elim_parse_script_output(buf); + if (isnan(v)) { + XCECD_LOG(XCE_LOG_WARN, + "elim: command \"%s\" output did not parse as a value " + "(got %.60s)", st->command, buf); + if (!st->have_value) { + return strdup("0"); + } + return strdup(""); + } + + st->value = v; + st->have_value = true; + st->next_run = time(NULL) + st->interval; + + /* collect_load_indices converts per valtype (fvalue: atof+free) and + * owns the string from here. */ + out_str = (char*)malloc(64); + if (out_str == NULL) { + return NULL; + } + snprintf(out_str, 64, "%f", v); + + return out_str; +} + +/* One trampoline per possible slot; the getter table indexes the static + * state array. (The registry API carries no user pointer.) */ +static char* elim_getter_0(void) { return elim_getter_run(0); } +static char* elim_getter_1(void) { return elim_getter_run(1); } +static char* elim_getter_2(void) { return elim_getter_run(2); } +static char* elim_getter_3(void) { return elim_getter_run(3); } +static char* elim_getter_4(void) { return elim_getter_run(4); } +static char* elim_getter_5(void) { return elim_getter_run(5); } +static char* elim_getter_6(void) { return elim_getter_run(6); } +static char* elim_getter_7(void) { return elim_getter_run(7); } +static char* elim_getter_8(void) { return elim_getter_run(8); } +static char* elim_getter_9(void) { return elim_getter_run(9); } +static char* elim_getter_10(void) { return elim_getter_run(10); } +static char* elim_getter_11(void) { return elim_getter_run(11); } +static char* elim_getter_12(void) { return elim_getter_run(12); } +static char* elim_getter_13(void) { return elim_getter_run(13); } +static char* elim_getter_14(void) { return elim_getter_run(14); } +static char* elim_getter_15(void) { return elim_getter_run(15); } + +static char* (*const g_elim_getters[])(void) = { + elim_getter_0, elim_getter_1, elim_getter_2, elim_getter_3, + elim_getter_4, elim_getter_5, elim_getter_6, elim_getter_7, + elim_getter_8, elim_getter_9, elim_getter_10, elim_getter_11, + elim_getter_12, elim_getter_13, elim_getter_14, elim_getter_15, +}; + +#define ELIM_MAX_SOURCES ((int)(sizeof(g_elim_getters) / sizeof(g_elim_getters[0]))) + +/* There is exactly one getter call site -- collect_load_indices() on the + * sync thread -- but registration happens on main before that thread + * exists; a mutex keeps the state array vs. getter table relationship + * honest without serializing the runs themselves. */ +int elim_register_sources(const elim_config_t* cfg) +{ + int i; + int rc = 0; + + if ((cfg == NULL) || (cfg->nsrc < 0) + || ((cfg->nsrc > 0) && (cfg->src == NULL))) { + return -EINVAL; + } + + if (cfg->nsrc == 0) { + return 0; + } + + if (cfg->nsrc > ELIM_MAX_SOURCES) { + XCECD_LOG(XCE_LOG_ERROR, + "elim: %d external resources configured, at most %d " + "supported", cfg->nsrc, ELIM_MAX_SOURCES); + return -EINVAL; + } + + pthread_mutex_lock(&g_elim_mutex); + + /* The 11 builtins register lazily at collect time; an elim source named + * "mem" registered before them would win the name race and SHADOW a + * builtin (the builtin's own registration then bounces as a dup). + * Force the builtin init first, so a name collision is a clean -EEXIST + * here, at startup, where the operator sees it. */ + resource_sync_ensure_builtin_load_indices(); + + g_elim_states = (elim_source_state_t*)calloc( + (size_t)cfg->nsrc, sizeof(elim_source_state_t)); + if (g_elim_states == NULL) { + pthread_mutex_unlock(&g_elim_mutex); + return -ENOMEM; + } + g_elim_nstates = cfg->nsrc; + + for (i = 0; i < cfg->nsrc; i++) { + if (!resource_sync_register_load_index( + cfg->src[i].name, Value_u_fvalue, g_elim_getters[i], false)) { + /* Duplicate name (another source, or one of the 11 builtins -- + * a user script must not shadow "mem"/"r15s"/...): surface the + * misconfiguration at startup, not as a silent missing value. */ + XCECD_LOG(XCE_LOG_ERROR, + "elim: cannot register external resource '%s' -- " + "name already in use", + cfg->src[i].name); + rc = -EEXIST; + break; + } + + g_elim_states[i].command = strdup(cfg->src[i].command); + if (g_elim_states[i].command == NULL) { + resource_sync_unregister_load_index(cfg->src[i].name); + rc = -ENOMEM; + break; + } + g_elim_states[i].interval = cfg->src[i].interval; + g_elim_states[i].have_value = false; + g_elim_states[i].next_run = 0; + + XCECD_LOG(XCE_LOG_INFO, + "elim: registered external resource '%s' (command: %s, " + "interval: %ds)", + cfg->src[i].name, g_elim_states[i].command, + g_elim_states[i].interval); + } + + pthread_mutex_unlock(&g_elim_mutex); + + if (rc < 0) { + elim_unregister_sources(cfg); + } + + return rc; +} + +void elim_unregister_sources(const elim_config_t* cfg) +{ + int i; + + if ((cfg == NULL) || (cfg->src == NULL)) { + return; + } + + pthread_mutex_lock(&g_elim_mutex); + + for (i = 0; i < cfg->nsrc; i++) { + resource_sync_unregister_load_index(cfg->src[i].name); + } + + for (i = 0; i < g_elim_nstates; i++) { + free(g_elim_states[i].command); + } + free(g_elim_states); + g_elim_states = NULL; + g_elim_nstates = 0; + + pthread_mutex_unlock(&g_elim_mutex); +} diff --git a/src/worker/src/main.c b/src/worker/src/main.c index bd27e9d6..e00f5656 100644 --- a/src/worker/src/main.c +++ b/src/worker/src/main.c @@ -394,10 +394,12 @@ int main(int argc, char* argv[]) char logfile[1024] = { 0 }; char* logdir = NULL; + const char* config_file = NULL; static struct option long_opts[] = { { "peers", required_argument, NULL, OPT_PEERS }, { "version", no_argument, NULL, OPT_VERSION }, + { "config", required_argument, NULL, 'c' }, { "help", no_argument, NULL, 'h' }, { NULL, 0, NULL, 0 } }; @@ -418,12 +420,15 @@ int main(int argc, char* argv[]) STR(VERSION_BUILD_TIME), STR(VERSION_DIRTY_FINGERPRINT)); - while ((opt = getopt_long(argc, argv, "n:l:vh", long_opts, NULL)) + while ((opt = getopt_long(argc, argv, "n:l:c:vh", long_opts, NULL)) != -1) { switch (opt) { case 'n': config.virutal_name = optarg; break; + case 'c': + config_file = optarg; + break; case 'l': if (optarg) { if (logdir) { @@ -450,9 +455,13 @@ int main(int argc, char* argv[]) return 0; case 'h': default: - printf("Usage: %s [-n name] [-l log_dir] [-v] " + printf("Usage: %s [-n name] [-l log_dir] [-c config] [-v] " "[--peers ctrl-1=host:port,...] " "[-h help]\n" + " -c FILE elim collector config (external " + "resources reported by\n" + " scripts on this host; design " + "external-license-resources §3.3b)\n" " --peers LIST HA controller candidate list: " "[name=]host:port,...\n" " When omitted, the XCE_CONTROLLER_PEERS " @@ -638,6 +647,41 @@ int main(int argc, char* argv[]) } #endif + /* elim collectors (design §3.3b): load the external_resources config + * before xcecd_init() registers its sources. No -c (or a file without + * the section) means zero external resources -- plain worker behavior. + * A config that is present but BROKEN stops startup loudly: a worker + * whose operator asked for elim reporting must not silently run + * without it (same fail-loud-at-load rule as esd's config). */ + if (config_file != NULL) { + elim_config_t elim_cfg; + int erc = elim_config_load(config_file, &elim_cfg); + + if (erc == -ENOENT) { + XCECD_LOG(XCE_LOG_ERROR, + "elim config file not found: %s", config_file); + fprintf(stderr, "xcecd: elim config file not found: %s\n", + config_file); + FREEUP(logdir); + return 1; + } + if (erc < 0) { + XCECD_LOG(XCE_LOG_ERROR, + "invalid elim config file %s (rc=%d) -- a source " + "needs both name and command", config_file, erc); + fprintf(stderr, + "xcecd: invalid elim config file %s (rc=%d)\n", + config_file, erc); + FREEUP(logdir); + return 1; + } + config.elim.nsrc = elim_cfg.nsrc; + config.elim.src = elim_cfg.src; /* ownership -> xcecd_config_t */ + XCECD_LOG(XCE_LOG_INFO, + "elim: %d external resource(s) from %s", + config.elim.nsrc, config_file); + } + if (xcecd_config_init(&config) != 0) { XCECD_LOG(XCE_LOG_ERROR, "Failed to initialize daemon configuration."); return 1; diff --git a/src/worker/src/resource_sync.c b/src/worker/src/resource_sync.c index 18666005..3e08424d 100644 --- a/src/worker/src/resource_sync.c +++ b/src/worker/src/resource_sync.c @@ -77,8 +77,17 @@ static void init_builtin_load_indices(void) * on resource_sync_thread()'s single dedicated pthread in production, * or single-threaded test code). */ g_load_indices_capacity = MAX_LOAD_INDICES; - g_load_indices = (load_index_info_t*)calloc(g_load_indices_capacity, - sizeof(load_index_info_t)); + if (g_load_indices == NULL) { + g_load_indices = (load_index_info_t*)calloc(g_load_indices_capacity, + sizeof(load_index_info_t)); + } + /* Never realloc over a populated array: since resource_sync_ensure_ + * builtin_load_indices() this function also runs with the array already + * alive (external entries registered first -- elim), and replacing it + * leaked the old one while leaving g_load_indices_count pointing past + * the fresh calloc (strcmp on garbage slots, segfault). The dup check + * in resource_sync_register_load_index() makes re-running the + * registration list idempotent. */ // register built-in load indices resource_sync_register_load_index("r15s", Value_u_fvalue, node_info_get_15s, @@ -105,6 +114,28 @@ static void init_builtin_load_indices(void) true); } +/** + * @brief Ensures the registry exists AND the 11 builtin load indices are + * registered in it, whichever external module (elim) queries or + * registers first. Idempotent; single-threaded callers only in + * practice (startup registration / the sync thread's collect), but + * safe under the registry mutex discipline its own callers keep. + */ +void resource_sync_ensure_builtin_load_indices(void) +{ + if (g_load_indices == NULL) { + init_builtin_load_indices(); + return; + } + /* init_builtin_load_indices() ran but the array was later torn down + * (resource_sync_destroy's cleanup) and a register-before-init caller + * re-populated it with external entries only -- the builtins must go + * in before the first report. */ + if (g_load_indices_count < BUILTIN_LOAD_INDICES_COUNT) { + init_builtin_load_indices(); + } +} + /** * @brief Registers a new load index. * @param name The name of the load index. @@ -123,6 +154,27 @@ bool resource_sync_register_load_index(const char* name, int valtype, pthread_mutex_lock(&g_load_indices_mutex); + /* elim (design §3.3b): external registration happens at daemon startup + * -- BEFORE the sync thread's first collect_load_indices() lazily + * allocated the array. A capacity-0 registry here used to reject every + * such registration with "Maximum load indices capacity reached (0)", + * silently dropping every configured external resource. Allocate the + * registry on first use, same as the lazy builtin init does; init_ + * builtin_load_indices() itself stays where it is (collect-time) so the + * 11 builtins keep registering exactly once per populated-registry + * window (bug-1756's contract -- its only caller's NULL check). */ + if (g_load_indices == NULL) { + g_load_indices_capacity = MAX_LOAD_INDICES; + g_load_indices = (load_index_info_t*)calloc( + g_load_indices_capacity, sizeof(load_index_info_t)); + if (g_load_indices == NULL) { + XCECD_LOG(XCE_LOG_ERROR, "Failed to allocate load indices array"); + g_load_indices_capacity = 0; + pthread_mutex_unlock(&g_load_indices_mutex); + return false; + } + } + for (int i = 0; i < g_load_indices_count; i++) { if (strcmp(g_load_indices[i].name, name) == 0) { XCECD_LOG(XCE_LOG_WARN, "Load index '%s' already exists", name); @@ -194,9 +246,7 @@ bool resource_sync_unregister_load_index(const char* name) static xce_resval_t* collect_load_indices(int* count) { - if (g_load_indices == NULL) { - init_builtin_load_indices(); - } + resource_sync_ensure_builtin_load_indices(); pthread_mutex_lock(&g_load_indices_mutex); diff --git a/src/worker/src/xcecd.c b/src/worker/src/xcecd.c index 466656eb..5a8cdb58 100644 --- a/src/worker/src/xcecd.c +++ b/src/worker/src/xcecd.c @@ -395,6 +395,22 @@ int xcecd_init(const xcecd_config_t* config) * xcecd_start() already ran. */ xcecd_ctx.cluster_cfg = config->cluster_cfg; + /* elim collectors (external-license-resources design §3.3b): register + * the configured external resources as custom load indices BEFORE the + * resource_sync thread starts (xcecd_start() -> resource_sync_start()), + * so the first 15-tick report already carries them. Ownership of + * config->elim moves here (config is read-only). */ + xcecd_ctx.elim.nsrc = config->elim.nsrc; + xcecd_ctx.elim.src = config->elim.src; + if (xcecd_ctx.elim.nsrc > 0) { + if (elim_register_sources(&xcecd_ctx.elim) < 0) { + XCECD_LOG(XCE_LOG_ERROR, + "Failed to register elim external resources -- " + "check external_resources config"); + goto cleanup_remote_host; + } + } + /* thread pool initializing */ xcecd_ctx.thread_pool = thread_pool_create(config->thread_pool_size); if (!xcecd_ctx.thread_pool) { @@ -457,6 +473,7 @@ cleanup_process_manager: cleanup_thread_pool: thread_pool_destroy(xcecd_ctx.thread_pool); cleanup_remote_host: + elim_unregister_sources(&xcecd_ctx.elim); free(xcecd_ctx.remote_host); free(xcecd_ctx.work_dir); free(xcecd_ctx.db_path); @@ -575,6 +592,9 @@ void xcecd_cleanup(void) resource_sync_stop(xcecd_ctx.resource_sync); resource_sync_destroy(xcecd_ctx.resource_sync); } + /* elim teardown AFTER resource_sync_destroy (which tears the registry + * down anyway -- this frees the source state and command strings). */ + elim_unregister_sources(&xcecd_ctx.elim); if (xcecd_ctx.db) { xcecd_db_close(xcecd_ctx.db); } diff --git a/src/worker/src/xcecd_cgroups.c b/src/worker/src/xcecd_cgroups.c index e3c84e66..78321ca5 100644 --- a/src/worker/src/xcecd_cgroups.c +++ b/src/worker/src/xcecd_cgroups.c @@ -1450,7 +1450,12 @@ xce_job_rusages_t* collect_job_rusages(uint64_t job_id) return NULL; } - snprintf(unit_name, sizeof(unit_name), "xcecd-worker-%lu.service", job_id); + /* Build the unit name the same way start_systemd_job() did (four-segment + * fmt_jobid_ex form) -- a bare %lu of the packed jobid never matches the + * real unit, so every property lookup below missed and all four counters + * stayed 0 (merge cb0c7fe9 regression). */ + make_job_sd_unit_name(unit_name, sizeof(unit_name) - 1, "xcecd-worker", + job_id); char* escaped_unit_name = replace_chars(unit_name); if (!escaped_unit_name) { XCECD_LOG(XCE_LOG_ERROR, "Failed to escape unit name."); diff --git a/src/worker/src/xcecd_exdata.c b/src/worker/src/xcecd_exdata.c index 78e6163c..f68edf4f 100644 --- a/src/worker/src/xcecd_exdata.c +++ b/src/worker/src/xcecd_exdata.c @@ -452,8 +452,12 @@ void apply_env_vars(env_var_t** env_vars, uint64_t jobid) XCECD_WORKER_LOG(XCE_LOG_DEBUG, "Applying environment variables..."); // Set some default XCE environment variables - char jobid_str[32]; - snprintf(jobid_str, sizeof(jobid_str), "%lu", jobid); + // XCE_JOBID is the four-segment written identity (cluster-apiserver- + // timestamp-sequence, fmt_jobid) -- a bare decimal of the packed uint64 + // is unusable in scripts (output-file names, xcecli job selection all + // speak the written form; same convention as every log line). + char jobid_str[64]; + fmt_jobid(jobid_str, sizeof(jobid_str), jobid); if (setenv("XCE_JOBID", jobid_str, 1) != 0) { XCECD_WORKER_LOG(XCE_LOG_ERROR, "Error setting environment variable: %s", diff --git a/src/worker/src/xrun.c b/src/worker/src/xrun.c index 6006dbb4..c86baeec 100644 --- a/src/worker/src/xrun.c +++ b/src/worker/src/xrun.c @@ -69,7 +69,16 @@ int xrun_ipc_send_remote_start(const char* hostname, const char* command) char* jobid_str = getenv("XCE_JOBID"); if (jobid_str) { - request_msg.data.remote_start.jobid = strtoull(jobid_str, NULL, 10); + /* XCE_JOBID is the four-segment written identity (fmt_jobid); + * parse_jobid() accepts exactly that form (a bare decimal of the + * packed uint64 no longer round-trips through the written form). */ + if (parse_jobid(&request_msg.data.remote_start.jobid, jobid_str) + != 1) { + XCECD_WORKER_LOG(XCE_LOG_WARN, + "xrun_ipc: cannot parse XCE_JOBID '%s'; " + "remote start will not match the job", + jobid_str); + } } if (send(sock_fd, &request_msg, sizeof(request_msg), 0) < 0) { diff --git a/src/worker/testsuite/meson.build b/src/worker/testsuite/meson.build index 2464387f..bca9b9f8 100644 --- a/src/worker/testsuite/meson.build +++ b/src/worker/testsuite/meson.build @@ -45,6 +45,23 @@ test('worker resource_sync test', resource_sync_test, timeout: 60, env: ['ASAN_OPTIONS=detect_leaks=0']) +# external-license-resources design §3.3b: the elim collector -- a +# config-driven script runner (posix_spawn + watchdog) that registers +# user-defined per-host load indices into resource_sync's registry at +# worker startup. White-box #includes elim.c AND resource_sync.c (the +# registry capacity-0-before-init trap is only observable against the real +# registry); see test_elim.c's header comment. +elim_test = executable('elim_test', + 'test_elim.c', + include_directories: [worker_test_incdir], + dependencies: worker_test_deps, + install: false) + +test('worker elim test', elim_test, + suite: 'worker', + timeout: 60, + env: ['ASAN_OPTIONS=detect_leaks=0']) + # bug-1554: real maxswap (sysinfo() totalswap)/maxtmp (statvfs("/tmp")) # detection -- white-box #include of hardware_monitor.c, same convention as # resource_sync_test above. diff --git a/src/worker/testsuite/test_elim.c b/src/worker/testsuite/test_elim.c new file mode 100644 index 00000000..ebfa1a93 --- /dev/null +++ b/src/worker/testsuite/test_elim.c @@ -0,0 +1,566 @@ +/* + * test_elim.c -- elim collector module (external-license-resources design + * §3.3b): xcecd gains a config-driven script collector that registers + * user-defined load indices into resource_sync's registry (the previously + * dead external-register API), so per-host "external resources" reported by + * user scripts on each host flow the live load-indices push chain + * (collect_load_indices -> xcecd_send_load_indices -> controller + * update_val_res -> scheduler RESVALS_DELTA -> sched_check_host_satisfy_). + * + * White-box #include of ../src/elim.c (same convention as + * test_resource_sync.c) plus ../src/esd_script-shaped spawn code lives in + * elim.c itself; the config parse is exercised against real temp files + * (esd_test's mkdtemp pattern). The spawn/watchdog half IS unit tested here + * (unlike esd's test, which deferred it to E2E): a real /bin/sh child with + * a real timeout kill is deterministic and cheap, and the watchdog is the + * one piece of elim.c that can hang the sync thread if it regresses. + * + * resource_sync.c is ALSO #included (after elim.c) to exercise the real + * registry rather than a stub: the capacity-0-before-init trap this phase + * fixes (elim registers at startup, before the sync thread's first collect + * lazily allocated the array) is only observable against the real code. + */ +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "resource_sync.h" +#include "xcecd.h" + +/* Real globals the white-box resource_sync.c references (test_resource_sync.c + * convention). */ +xcecd_context_t xcecd_ctx; + +/* Stubs for the 11 builtin load-index getters (node_information.c, not + * linked here). */ +static char* stub_getter(void) +{ + return strdup("1.0"); +} + +char* node_info_get_15s(void) { return stub_getter(); } +char* node_info_get_1m(void) { return stub_getter(); } +char* node_info_get_15m(void) { return stub_getter(); } +char* node_info_get_ut(void) { return stub_getter(); } +char* node_info_get_pg(void) { return stub_getter(); } +char* node_info_get_ls(void) { return stub_getter(); } +char* node_info_get_it(void) { return stub_getter(); } +char* node_info_get_swap(void) { return stub_getter(); } +char* node_info_get_mem(void) { return stub_getter(); } +char* node_info_get_tmp(void) { return stub_getter(); } +char* node_info_get_io(void) { return stub_getter(); } + +/* Spy on the send (controller_connector.c, not linked here). */ +static int g_send_load_indices_call_count = 0; + +bool xcecd_send_load_indices(xce_resval_t** vals, int count) +{ + (void)vals; + (void)count; + g_send_load_indices_call_count++; + return true; +} + +/* resource_sync.c's resync gate (xcecd_resync.c, not linked here). */ +static bool g_test_resync_pending = false; + +bool xcecd_resync_is_pending(void) { return g_test_resync_pending; } + +#include "../src/elim.c" +#include "../src/resource_sync.c" + +static char g_tmpdir[256]; + +void setUp(void) +{ + snprintf(g_tmpdir, sizeof(g_tmpdir), "/tmp/xce_elim_testXXXXXX"); + TEST_ASSERT_NOT_NULL(mkdtemp(g_tmpdir)); + + memset(&xcecd_ctx, 0, sizeof(xcecd_ctx)); + xcecd_ctx.virtual_name = (char*)"t1"; + g_test_resync_pending = false; + g_send_load_indices_call_count = 0; +} + +void tearDown(void) +{ + /* Drop any registrations this test left so the next test starts from a + * clean registry: resource_sync_destroy on a throwaway object runs + * resource_sync_cleanup_load_indices() (static in the #included + * resource_sync.c, reachable only through this public entry). */ + resource_sync_destroy(resource_sync_create()); + + if (g_tmpdir[0] != '\0') { + char path[300]; + + snprintf(path, sizeof(path), "rm -rf %s", g_tmpdir); + if (system(path) != 0) { + /* best effort */ + } + g_tmpdir[0] = '\0'; + } +} + +static void write_file(const char* dir, const char* name, const char* content) +{ + char path[512]; + FILE* f; + + snprintf(path, sizeof(path), "%s/%s", dir, name); + f = fopen(path, "w"); + TEST_ASSERT_NOT_NULL(f); + fputs(content, f); + fclose(f); +} + +/* ── elim config parse ────────────────────────────────────────────────── */ + +static void test_elim_config_load_full_file(void) +{ + elim_config_t cfg; + int rc; + + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: dynres1\n" + " command: /opt/xce/scripts/report-dynres1.sh\n" + " interval: 30\n" + " - name: gpu_free\n" + " command: /usr/local/bin/gpu-free.sh\n" + " interval: 60\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(2, cfg.nsrc); + TEST_ASSERT_EQUAL_STRING("dynres1", cfg.src[0].name); + TEST_ASSERT_EQUAL_STRING("/opt/xce/scripts/report-dynres1.sh", + cfg.src[0].command); + TEST_ASSERT_EQUAL_INT(30, cfg.src[0].interval); + TEST_ASSERT_EQUAL_STRING("gpu_free", cfg.src[1].name); + TEST_ASSERT_EQUAL_INT(60, cfg.src[1].interval); + + elim_config_fini(&cfg); +} + +static void test_elim_config_interval_defaults_when_missing(void) +{ + elim_config_t cfg; + int rc; + + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: dynres1\n" + " command: /opt/xce/scripts/r.sh\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(1, cfg.nsrc); + /* 0 = "every send tick" -- the getter cache refreshes each time the + * sync thread collects. */ + TEST_ASSERT_EQUAL_INT(0, cfg.src[0].interval); + + elim_config_fini(&cfg); +} + +static void test_elim_config_missing_file_is_enoent(void) +{ + elim_config_t cfg; + int rc; + + rc = elim_config_load_path(g_tmpdir, "no-such-file.yaml", &cfg); + + TEST_ASSERT_EQUAL_INT(-ENOENT, rc); +} + +static void test_elim_config_missing_command_is_einval(void) +{ + elim_config_t cfg; + int rc; + + /* A source without a command can never report -- fail loudly at load, + * not silently at runtime (esd_config.c's validation rule). */ + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: dynres1\n" + " interval: 30\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + + TEST_ASSERT_EQUAL_INT(-EINVAL, rc); +} + +static void test_elim_config_missing_name_is_einval(void) +{ + elim_config_t cfg; + int rc; + + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - command: /opt/xce/scripts/r.sh\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + + TEST_ASSERT_EQUAL_INT(-EINVAL, rc); +} + +static void test_elim_config_empty_or_absent_section_is_ok(void) +{ + elim_config_t cfg; + int rc; + + /* A worker config with no external_resources key is the normal case for + * every host without elim scripts -- must load to zero sources, not + * error. */ + write_file(g_tmpdir, "elim.yaml", "external_resources: []\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_EQUAL_INT(0, cfg.nsrc); + TEST_ASSERT_NULL(cfg.src); + + elim_config_fini(&cfg); +} + +/* ── elim output parse ────────────────────────────────────────────────── */ + +static void test_elim_parse_output_name_value_line(void) +{ + double v; + + /* The elim contract: first non-empty line carries " ". */ + v = elim_parse_script_output("dynres1 42\n"); + TEST_ASSERT_FLOAT_WITHIN(1e-6, 42.0, v); + + v = elim_parse_script_output("\n\n# comment-ish noise 7 8\ngpu_free 5.5\n"); + TEST_ASSERT_FLOAT_WITHIN(1e-6, 7.0, v); +} + +static void test_elim_parse_output_bare_value_line(void) +{ + double v; + + v = elim_parse_script_output("12\n"); + TEST_ASSERT_FLOAT_WITHIN(1e-6, 12.0, v); + + v = elim_parse_script_output(" 3.5 \n"); + TEST_ASSERT_FLOAT_WITHIN(1e-6, 3.5, v); +} + +static void test_elim_parse_output_garbage_is_nan(void) +{ + TEST_ASSERT(isnan(elim_parse_script_output("no numbers here\n"))); + TEST_ASSERT(isnan(elim_parse_script_output(""))); + TEST_ASSERT(isnan(elim_parse_script_output(NULL))); +} + +/* ── registry wiring: register at startup, before any collect ─────────── */ + +/* THE trap this phase fixes: before it, resource_sync_register_load_index + * called at daemon startup (when g_load_indices is still NULL -- the array + * is only allocated by collect_load_indices' lazy init_builtin_load_indices, + * i.e. on the sync thread's first 15th tick) failed with "capacity reached + * (0)" and every elim source silently vanished. Registration must + * self-initialize. */ +static void test_register_load_index_before_lazy_init_succeeds(void) +{ + bool ok; + + /* No collect_load_indices() call ever happened in this process yet. */ + TEST_ASSERT_NULL_MESSAGE(g_load_indices, + "precondition: registry not yet initialized"); + + ok = resource_sync_register_load_index("dynres1", Value_u_fvalue, + stub_getter, false); + TEST_ASSERT_TRUE_MESSAGE( + ok, "registering before the lazy builtin init must not fail with " + "'capacity reached (0)'"); + + /* And the value must actually flow: collect_load_indices now includes + * the elim index among the builtins. */ + xce_resval_t* vals; + int count = 0; + int found = 0; + + vals = collect_load_indices(&count); + TEST_ASSERT_NOT_NULL(vals); + for (int i = 0; i < count; i++) { + if (strcmp(vals[i].name, "dynres1") == 0) { + found++; + TEST_ASSERT_EQUAL_STRING("t1", vals[i].host); + TEST_ASSERT_EQUAL_INT(Value_u_fvalue, vals[i].val->which); + TEST_ASSERT_FLOAT_WITHIN(1e-6, 1.0, vals[i].val->u.fvalue); + } + /* The host tag is the join identity for every report (bug-1569). */ + TEST_ASSERT_EQUAL_STRING("t1", vals[i].host); + } + TEST_ASSERT_EQUAL_INT(1, found); + + free_resvals(vals, count); +} + +static void test_elim_sources_register_and_flow_through_registry(void) +{ + elim_config_t cfg; + int rc; + + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: dynres1\n" + " command: echo dynres1 7\n" + " interval: 0\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + + rc = elim_register_sources(&cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + + /* The getter runs the script: first collect refreshes the cache. */ + xce_resval_t* vals; + int count = 0; + int found = 0; + + vals = collect_load_indices(&count); + TEST_ASSERT_NOT_NULL(vals); + for (int i = 0; i < count; i++) { + if (strcmp(vals[i].name, "dynres1") == 0) { + found++; + TEST_ASSERT_EQUAL_INT(Value_u_fvalue, vals[i].val->which); + /* "echo dynres1 7" -> name-value line -> 7.0 */ + TEST_ASSERT_FLOAT_WITHIN(1e-6, 7.0, vals[i].val->u.fvalue); + } + } + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, found, "the elim-registered index must reach collect_load_indices"); + + free_resvals(vals, count); +} + +static void test_elim_register_rejects_duplicate_name(void) +{ + elim_config_t cfg; + int rc; + + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: dynres1\n" + " command: echo dynres1 1\n" + " - name: dynres1\n" + " command: echo dynres1 2\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + + /* The registry rejects the dup; elim_register_sources reports the + * failure so the operator sees the misconfiguration at startup. */ + TEST_ASSERT_EQUAL_INT(-EEXIST, elim_register_sources(&cfg)); +} + +static void test_elim_register_rejects_builtin_name_override(void) +{ + elim_config_t cfg; + int rc; + + /* "mem" is one of the 11 builtins -- a user script must not shadow it + * (the registry rejects the dup; the error surfaces at startup). */ + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: mem\n" + " command: echo mem 999\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + + TEST_ASSERT_EQUAL_INT(-EEXIST, elim_register_sources(&cfg)); +} + +static void test_elim_unregister_sources_cleans_up(void) +{ + elim_config_t cfg; + int rc; + + write_file(g_tmpdir, "elim.yaml", + "external_resources:\n" + " - name: dynres1\n" + " command: echo dynres1 1\n"); + + rc = elim_config_load_path(g_tmpdir, "elim.yaml", &cfg); + TEST_ASSERT_EQUAL_INT(0, rc); + + TEST_ASSERT_EQUAL_INT(0, elim_register_sources(&cfg)); + TEST_ASSERT_TRUE(resource_sync_unregister_load_index("dynres1")); + TEST_ASSERT_FALSE(resource_sync_unregister_load_index("dynres1")); +} + +/* ── the script runner: real child, real watchdog ─────────────────────── */ + +static void test_elim_script_collect_runs_command(void) +{ + char out[128]; + int rc; + const char* script; + + script = "/bin/sh -c 'echo dynres1 42'"; + rc = elim_script_collect(script, out, sizeof(out)); + TEST_ASSERT_EQUAL_INT(0, rc); + TEST_ASSERT_FLOAT_WITHIN(1e-6, 42.0, elim_parse_script_output(out)); +} + +static void test_elim_script_collect_timeout_kills_child(void) +{ + char out[128]; + int rc; + int saved_timeout; + const char* script; + + /* A script that hangs must be killed at the watchdog deadline, not + * stall the sync thread forever (the esd_script watchdog contract). */ + script = "/bin/sh -c 'sleep 60'"; + saved_timeout = elim_script_timeout_ms; + elim_script_timeout_ms = 500; + + rc = elim_script_collect(script, out, sizeof(out)); + + elim_script_timeout_ms = saved_timeout; + + TEST_ASSERT_EQUAL_INT_MESSAGE(-ETIMEDOUT, rc, + "a hung script must hit the watchdog"); + + /* And the child must actually be reaped -- no zombie left behind. */ + TEST_ASSERT_EQUAL_INT_MESSAGE(-1, waitpid(-1, NULL, WNOHANG), + "no unreaped child may survive the kill"); +} + +static void test_elim_script_collect_empty_output_is_enodata(void) +{ + char out[128]; + int rc; + + rc = elim_script_collect("/bin/sh -c 'exit 0'", out, sizeof(out)); + TEST_ASSERT_EQUAL_INT(-ENODATA, rc); +} + +/* ── getter cache semantics: interval ─────────────────────────────────── */ + +static void test_elim_getter_refreshes_per_interval(void) +{ + elim_config_t cfg; + xce_resval_t* vals; + int count = 0; + static char counter_script[512]; + static char yaml[1024]; + int v1 = -1, v2 = -1; + resource_sync_t* sync; + + /* A script whose output changes on every run: the getter must return + * the FRESH value after the interval elapses (refresh), and the cached + * one before it (no re-run). */ + snprintf(counter_script, sizeof(counter_script), "%s/counter.sh", g_tmpdir); + { + FILE* f = fopen(counter_script, "w"); + TEST_ASSERT_NOT_NULL(f); + fprintf(f, + "#!/bin/sh\n" + "if [ -f %s.n ]; then n=$(cat %s.n); n=$((n+1)); else n=1; fi\n" + "echo $n > %s.n\n" + "echo \"dynres1 $n\"\n", + counter_script, counter_script, counter_script); + fclose(f); + chmod(counter_script, 0755); + } + + snprintf(yaml, sizeof(yaml), + "external_resources:\n" + " - name: dynres1\n" + " command: %s\n" + " interval: 3\n", + counter_script); + write_file(g_tmpdir, "elim.yaml", yaml); + + TEST_ASSERT_EQUAL_INT(0, elim_config_load_path(g_tmpdir, "elim.yaml", &cfg)); + TEST_ASSERT_EQUAL_INT(0, elim_register_sources(&cfg)); + + /* First collect: getter runs the script (counter=1). */ + vals = collect_load_indices(&count); + TEST_ASSERT_NOT_NULL(vals); + for (int i = 0; i < count; i++) { + if (strcmp(vals[i].name, "dynres1") == 0) { + v1 = (int)vals[i].val->u.fvalue; + } + } + free_resvals(vals, count); + TEST_ASSERT_EQUAL_INT(1, v1); + + /* Collect again immediately (interval=3 not yet elapsed): the cached + * value is returned, the counter script does NOT advance. */ + vals = collect_load_indices(&count); + TEST_ASSERT_NOT_NULL(vals); + for (int i = 0; i < count; i++) { + if (strcmp(vals[i].name, "dynres1") == 0) { + v2 = (int)vals[i].val->u.fvalue; + } + } + free_resvals(vals, count); + TEST_ASSERT_EQUAL_INT_MESSAGE( + 1, v2, "getter must serve the cached value within its interval"); + + /* After the interval elapses the getter re-runs the script. Drive it + * through a real resource_sync tick loop rather than sleeping 3s in + * wall-clock: seed the counter and run two 1s ticks. */ + sync = resource_sync_create(); + TEST_ASSERT_NOT_NULL(sync); + sync->load_report_counter = LOAD_INDICES_INTERVAL_DEFAULT - 1; + TEST_ASSERT_TRUE(resource_sync_start(sync)); + /* Two ticks cover the 1s sleep-per-tick of the sync thread; interval 3 + * seconds measured from the first run has now elapsed. */ + sleep(4); + + vals = collect_load_indices(&count); + TEST_ASSERT_NOT_NULL(vals); + for (int i = 0; i < count; i++) { + if (strcmp(vals[i].name, "dynres1") == 0) { + v2 = (int)vals[i].val->u.fvalue; + } + } + free_resvals(vals, count); + + resource_sync_stop(sync); + resource_sync_destroy(sync); + + TEST_ASSERT_EQUAL_INT_MESSAGE( + 2, v2, "getter must refresh the value once its interval elapses"); +} + +int main(void) +{ + UNITY_BEGIN(); + RUN_TEST(test_elim_config_load_full_file); + RUN_TEST(test_elim_config_interval_defaults_when_missing); + RUN_TEST(test_elim_config_missing_file_is_enoent); + RUN_TEST(test_elim_config_missing_command_is_einval); + RUN_TEST(test_elim_config_missing_name_is_einval); + RUN_TEST(test_elim_config_empty_or_absent_section_is_ok); + RUN_TEST(test_elim_parse_output_name_value_line); + RUN_TEST(test_elim_parse_output_bare_value_line); + RUN_TEST(test_elim_parse_output_garbage_is_nan); + RUN_TEST(test_register_load_index_before_lazy_init_succeeds); + RUN_TEST(test_elim_sources_register_and_flow_through_registry); + RUN_TEST(test_elim_register_rejects_duplicate_name); + RUN_TEST(test_elim_register_rejects_builtin_name_override); + RUN_TEST(test_elim_unregister_sources_cleans_up); + RUN_TEST(test_elim_script_collect_runs_command); + RUN_TEST(test_elim_script_collect_timeout_kills_child); + RUN_TEST(test_elim_script_collect_empty_output_is_enodata); + RUN_TEST(test_elim_getter_refreshes_per_interval); + return UNITY_END(); +} diff --git a/src/worker/testsuite/test_exdata_jobindex.c b/src/worker/testsuite/test_exdata_jobindex.c index b6f33773..bcff9cb1 100644 --- a/src/worker/testsuite/test_exdata_jobindex.c +++ b/src/worker/testsuite/test_exdata_jobindex.c @@ -6,7 +6,8 @@ * the submission's own variables. XCE_JOBINDEX must be the element number of * an array-tagged job ID (a one-element array included) and 0 for an * ordinary job; a submission-supplied XCE_JOBINDEX is ignored with a WARN - * naming the job and the variable; XCE_JOBID handling is unchanged. + * naming the job and the variable; XCE_JOBID is the four-segment written + * identity (fmt_jobid), unchanged in override behavior. * * White-box #include of ../src/xcecd_exdata.c (same convention as the other * tests in this suite). XCECD_WORKER_LOG is redefined to a capture hook @@ -203,20 +204,20 @@ static void test_similar_names_still_applied(void) unsetenv("xce_jobindex"); } -/* 6.5: XCE_JOBID is the decimal job ID; a submission's own XCE_JOBID still - * overrides it, as before, with no WARN. */ +/* 6.5: XCE_JOBID is the four-segment written job identity (fmt_jobid); a + * submission's own XCE_JOBID still overrides it, as before, with no WARN. */ static void test_jobid_unchanged(void) { env_var_t* vars = NULL; - char dec[32]; + char idstr[64]; - snprintf(dec, sizeof(dec), "%lu", (unsigned long)ARRAY_ELEMENT(4)); + fmt_jobid(idstr, sizeof(idstr), ARRAY_ELEMENT(4)); apply_env_vars(&vars, ARRAY_ELEMENT(4)); - expect_env("XCE_JOBID", dec); + expect_env("XCE_JOBID", idstr); - snprintf(dec, sizeof(dec), "%lu", (unsigned long)ORDINARY_JOB); + fmt_jobid(idstr, sizeof(idstr), ORDINARY_JOB); apply_env_vars(&vars, ORDINARY_JOB); - expect_env("XCE_JOBID", dec); + expect_env("XCE_JOBID", idstr); captured_count = 0; vars = push_var(vars, "XCE_JOBID", "user-value"); diff --git a/src/worker/testsuite/test_xcecd_startup_order.c b/src/worker/testsuite/test_xcecd_startup_order.c index ae2e4a62..06eab968 100644 --- a/src/worker/testsuite/test_xcecd_startup_order.c +++ b/src/worker/testsuite/test_xcecd_startup_order.c @@ -161,6 +161,17 @@ bool resource_sync_start(resource_sync_t* sync) } void resource_sync_stop(resource_sync_t* sync) { (void)sync; } +/* elim surface (elim.c, not linked here -- this test stubs every subsystem + * constructor the same way): xcecd_init() registers the configured sources + * (zero sources in this test's configs, so the real impl would no-op, but + * the symbol must resolve); xcecd_cleanup() unregisters. */ +int elim_register_sources(const elim_config_t* cfg) +{ + (void)cfg; + return 0; +} +void elim_unregister_sources(const elim_config_t* cfg) { (void)cfg; } + /* Referenced by xcecd_process_command()/xcecd_cleanup() -- never invoked at * runtime by this test (only xcecd_init()/xcecd_start() are called), but * their bodies are compiled unconditionally so the symbols must resolve. */ diff --git a/src/xceadmin/src/xceadmin/build.py b/src/xceadmin/src/xceadmin/build.py index edf33494..94799ec1 100644 --- a/src/xceadmin/src/xceadmin/build.py +++ b/src/xceadmin/src/xceadmin/build.py @@ -208,7 +208,9 @@ def _format_resource(r) -> str: ordtype = _RESOURCE_ORDTYPE_NAMES.get(r.ordtype, r.ordtype) consumable = bool(r.flags & mapping.RESOURCE_FLAG_CONSUMABLE) release = bool(r.flags & mapping.RESOURCE_FLAG_RELEASE) + builtin = bool(r.flags & mapping.RESOURCE_FLAG_BUILTIN) shared = bool(r.flags & mapping.RESOURCE_FLAG_SHARED) + external = bool(r.flags & mapping.RESOURCE_FLAG_EXTERNAL) parts = [f"name={r.name}", f"type={valtype}", f"order={ordtype}"] if r.desc: parts.append(f"desc={r.desc!r}") @@ -216,6 +218,13 @@ def _format_resource(r) -> str: parts.append(f"interval={r.interval}") parts.append(f"consumable={consumable}") parts.append(f"release={release}") + # builtin/external change the resource's semantics (whose values count, + # where they come from) -- always visible when set, never shown bare + # false: absent means the plain user-resource default. + if builtin: + parts.append("builtin=True") + if external: + parts.append("external=True") if shared: parts.append("shared=true") if len(r.distributions) > 0: diff --git a/src/xceadmin/src/xceadmin/cli.py b/src/xceadmin/src/xceadmin/cli.py index 471feabb..c4bb0a1f 100644 --- a/src/xceadmin/src/xceadmin/cli.py +++ b/src/xceadmin/src/xceadmin/cli.py @@ -27,11 +27,14 @@ traceback. from __future__ import annotations +import json import os +import time from collections.abc import Callable from datetime import datetime, timezone from importlib.metadata import PackageNotFoundError from importlib.metadata import version as _package_version +from pathlib import Path import click from click.core import ParameterSource @@ -43,6 +46,7 @@ from xcecommon.errors import ( SchedAdminError, TooManyRedirectsError, XceAdminError, + XceExtresStatusError, ) from xceadmin.build import build_cluster_config, dump_cluster_config, write_output from xceadmin.push import probe_config, push_config @@ -372,6 +376,126 @@ def scheduler_dump(file: str) -> None: click.echo(text, nl=False) # dump() already ends its own text with "\n" (5.20). +# --------------------------------------------------------------------------- +# extres status (external-license-resources Phase 2 item 4, design.md 3.5): +# read the status JSON xceesd writes on its own host and render it for the +# operator. A LOCAL-file command like scheduler dump -- no --controller, no +# wire: the daemon's status file is the observability surface until the +# PR-2 proxy formalizes a status channel (design.md 3.6; a temporary +# remote protocol would be redesigned out from under us by PR-2's +# provider-lease model). Run it on the daemon's host. +# --------------------------------------------------------------------------- + +# Where the deployed esd.yaml's status_file default points. Matches +# esd_config.c's status_file default and the deploy docs; overridable with +# --status-file (the t1 test cluster uses a /tmp path instead). +DEFAULT_STATUS_FILE = "/var/run/xce/esd-status.json" + + +def _format_extres_status(payload: dict, now: int | None = None) -> str: + """Renders one decoded esd status object as operator text. + + ``now`` pins the wall clock for deterministic tests (None = time.time()). + Rendering rules: + - daemon identity line: name + pid + - uptime and push counters on their own lines + - per-source rows: NAME value= age=s [-- STALE] or + "no value yet" when last_value is null. + """ + if now is None: + now = int(time.time()) + + lines = [ + f"daemon: {payload.get('daemon', 'xceesd')} (pid {payload.get('pid', '?')})", + ] + + started_at = payload.get("started_at", 0) + if isinstance(started_at, int) and started_at > 0: + uptime = max(now - started_at, 0) + started_str = datetime.fromtimestamp(started_at, tz=timezone.utc).strftime("%Y-%m-%dT%H:%M:%S") + lines.append(f"started at: {started_str} (up {uptime}s)") + + lines.append( + f"pushes: {payload.get('pushes_ok', 0)} ok, " + f"{payload.get('pushes_fail', 0)} failed" + ) + + sources = payload.get("sources", []) + if not sources: + lines.append("sources: (none configured)") + return "\n".join(lines) + + lines.append("sources:") + for src in sources: + name = src.get("name", "?") + if src.get("last_value") is None: + lines.append(f" {name}: no value yet") + continue + last_ok = src.get("last_ok_time", 0) + age = max(now - last_ok, 0) if isinstance(last_ok, int) and last_ok > 0 else 0 + stale = " -- STALE" if src.get("stale") else "" + lines.append(f" {name}: value={src.get('last_value')} age={age}s{stale}") + + return "\n".join(lines) + + +def _read_extres_status(path: str) -> dict: + """Reads and validates one esd status file. Raises + :class:`XceExtresStatusError` for every local failure mode (unreadable + path, unparseable JSON, JSON that is not an esd status object) so the + CLI layer never sees a raw OSError/JSONDecodeError. + """ + try: + raw = Path(path).read_text() + except OSError as exc: + raise XceExtresStatusError( + f"cannot read extres status file {path!r}: {exc}", path=path, reason=str(exc) + ) from exc + + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise XceExtresStatusError( + f"extres status file {path!r} is not valid JSON: {exc}", path=path, reason=str(exc) + ) from exc + + if not isinstance(payload, dict) or "sources" not in payload or "daemon" not in payload: + raise XceExtresStatusError( + f"extres status file {path!r} is not an xceesd status object " + "(missing 'daemon' or 'sources')", + path=path, + reason="wrong schema", + ) + return payload + + +@cli.group() +def extres() -> None: + """Inspect the external-resources daemon (xceesd).""" + + +@extres.command("status") +@click.option( + "--status-file", + default=None, + help=f"Path of the xceesd status JSON (default: {DEFAULT_STATUS_FILE}).", +) +def extres_status(status_file: str | None) -> None: + """Report xceesd's status from its local status file (run on the daemon's host).""" + # Resolved at call time, not decoration time, so tests (and wrappers) + # can re-point the default -- click freezes a non-None default= into + # the command's own default_value at decoration. + if status_file is None: + status_file = DEFAULT_STATUS_FILE + try: + payload = _read_extres_status(status_file) + click.echo(_format_extres_status(payload)) + except XceAdminError as e: + raise click.ClickException(e.message) from e + except KeyboardInterrupt: + raise click.ClickException("cancelled") from None + + def main() -> None: """Console-script entry point (see pyproject.toml [project.scripts]).""" cli() diff --git a/src/xceadmin/src/xcecommon/errors.py b/src/xceadmin/src/xcecommon/errors.py index 480af1c7..7bec4230 100644 --- a/src/xceadmin/src/xcecommon/errors.py +++ b/src/xceadmin/src/xcecommon/errors.py @@ -222,3 +222,23 @@ class SchedAdminError(XceAdminError): self.file = file self.event_count = event_count self.started_at = started_at + + +class XceExtresStatusError(XceAdminError): + """The xceesd status file (``xceadmin extres status``) could not be + read or decoded (external-license-resources Phase 2 item 4, design.md + 3.5; raised by ``xceadmin.cli._read_extres_status``). + + Covers every local-file failure mode: the path doesn't exist / isn't + readable, the bytes aren't valid JSON, or the JSON parses but isn't an + esd status object (missing ``daemon``/``sources`` -- e.g. someone + pointed ``--status-file`` at some other JSON file). Same local-file + concern as :class:`ReplayFileError`/:class:`ClusterBinFileError`: no + network framing involved -- ``extres status`` reads the status JSON + xceesd writes on its own host. + """ + + def __init__(self, message: str, path: str, reason: str) -> None: + super().__init__(message) + self.path = path + self.reason = reason diff --git a/src/xceadmin/src/xcecommon/mapping.py b/src/xceadmin/src/xcecommon/mapping.py index 007efa96..2637a373 100644 --- a/src/xceadmin/src/xcecommon/mapping.py +++ b/src/xceadmin/src/xcecommon/mapping.py @@ -64,12 +64,12 @@ reason -- never a silent drop. ``Resource`` (``resources.yaml``) ``valtype``/``ordtype``/``flags`` ARE mapped from YAML - ``type``/``order``/``consumable``/``release`` -- the encoding is - documented, just not in ``controller.capnp``/design.md: it lives in + ``type``/``order``/``consumable``/``release``/``builtin`` -- the encoding + is documented, just not in ``controller.capnp``/design.md: it lives in ``include/xce/types.h`` (``XCE_VAL_*``, ``xce_order_t``, - ``XCE_RESF_CONSUMABLE``/``XCE_RESF_RELEASE``) and is actively read at - runtime (e.g. ``scheduler.c`` copies these three fields straight out - of the parsed resource). See :data:`RESOURCE_VALTYPE`, + ``XCE_RESF_CONSUMABLE``/``XCE_RESF_RELEASE``/``XCE_RESF_BUILTIN``) and + is actively read at runtime (e.g. ``scheduler.c`` copies these three + fields straight out of the parsed resource). See :data:`RESOURCE_VALTYPE`, :data:`RESOURCE_ORDTYPE`, and the flag constants below for the exact values (also reused by ``xceadmin.build.dump_cluster_config`` to render these int-encoded fields back to their YAML names). ``idx`` and @@ -205,10 +205,16 @@ _XCE_SCALAR_FIELDS: tuple[tuple[str, str], ...] = ( RESOURCE_VALTYPE: dict[str, int] = {"string": 0, "numeric": 1, "boolean": 2} RESOURCE_ORDTYPE: dict[str, int] = {"increase": 0, "decrease": 1, "na": 2} -# resources.yaml ``consumable``/``release`` -> Resource.flags bits, mirroring -# include/xce/types.h's XCE_RESF_CONSUMABLE/XCE_RESF_RELEASE. +# resources.yaml ``consumable``/``release``/``builtin`` -> Resource.flags +# bits, mirroring include/xce/types.h's XCE_RESF_CONSUMABLE/ +# XCE_RESF_RELEASE/XCE_RESF_BUILTIN. RESOURCE_FLAG_CONSUMABLE = 0x80 RESOURCE_FLAG_RELEASE = 0x20 +# XCE_RESF_BUILTIN: the resource's values are produced by hardcoded worker/ +# controller reporters (resource_sync.c's load indices, worker.c's host +# attributes and slots) -- display/accounting code uses it to tell those +# from user-declared resources of the same name space. +RESOURCE_FLAG_BUILTIN = 0x01 # Set together on a resourcemap item whose location is exactly ["all"] # (cluster-wide static shared pool): XCE_RESF_SHARED | XCE_RESF_GLOBAL in @@ -221,6 +227,14 @@ RESOURCE_FLAG_SHARED = 0x08 RESOURCE_FLAG_GLOBAL = 0x04 RESOURCE_FLAG_SHARED_GLOBAL = RESOURCE_FLAG_SHARED | RESOURCE_FLAG_GLOBAL +# Set together with SHARED|GLOBAL on a resourcemap item whose location is +# exactly ["external"] (external-license-resources): XCE_RESF_EXTERNAL in +# include/xce/types.h. The pool is cluster-wide but its values come only +# from the extres daemon's free-count pushes (host=="" resval deltas) -- the +# controller must NOT seed it from a config value (seed_sharedinst_from_config +# skips EXTERNAL) and the scheduler's admission formula skips its inuse term. +RESOURCE_FLAG_EXTERNAL = 0x10 + def new_cluster_config_builder() -> "capnp._DynamicStructBuilder": @@ -413,6 +427,8 @@ def _build_resource(item: dict, target: "capnp._DynamicStructBuilder") -> None: flags |= RESOURCE_FLAG_CONSUMABLE if item.get("release"): flags |= RESOURCE_FLAG_RELEASE + if item.get("builtin"): + flags |= RESOURCE_FLAG_BUILTIN resourcemap = item.get("resourcemap") or [] if resourcemap: distributions = target.init("distributions", len(resourcemap)) @@ -429,6 +445,11 @@ def _build_resource(item: dict, target: "capnp._DynamicStructBuilder") -> None: # scheduler can admit jobs against today (see module # docstring, Resource). flags |= RESOURCE_FLAG_SHARED_GLOBAL + elif locations == ["external"]: + # Cluster-wide externally-reported pool + # (external-license-resources): daemon pushes the free + # values; config declares only the name/shape. + flags |= RESOURCE_FLAG_SHARED_GLOBAL | RESOURCE_FLAG_EXTERNAL target.flags = flags diff --git a/src/xceadmin/tests/fixtures/full/configs/resources.yaml b/src/xceadmin/tests/fixtures/full/configs/resources.yaml index f12b9f82..22be4e85 100644 --- a/src/xceadmin/tests/fixtures/full/configs/resources.yaml +++ b/src/xceadmin/tests/fixtures/full/configs/resources.yaml @@ -12,6 +12,7 @@ resources: type: numeric interval: 60 consumable: false + builtin: true order: decrease description: "available memory in MB (built-in dynamic load index)" @@ -197,3 +198,16 @@ resources: - name: gstring description: string resource type: string + + # external-license-resources: cluster-wide license pool reported by + # the extres daemon (xceesd) -- config declares only the name/shape, + # the free counts come from the daemon's pushes + - name: extlic + description: externally reported license pool + type: numeric + interval: 0 + consumable: true + order: decrease + resourcemap: + - location: + - external diff --git a/src/xceadmin/tests/fixtures/full/expected.bin b/src/xceadmin/tests/fixtures/full/expected.bin index 4d0e67af..3fa1e0a7 100644 Binary files a/src/xceadmin/tests/fixtures/full/expected.bin and b/src/xceadmin/tests/fixtures/full/expected.bin differ diff --git a/src/xceadmin/tests/test_cli_extres.py b/src/xceadmin/tests/test_cli_extres.py new file mode 100644 index 00000000..a8c4c8e6 --- /dev/null +++ b/src/xceadmin/tests/test_cli_extres.py @@ -0,0 +1,275 @@ +"""Tests for the ``extres`` command group (external-license-resources +Phase 2 item 4, design.md 3.5 "``xceadmin extres status``"). + +``extres status`` is a LOCAL-file command -- the inverse slice of +``scheduler dump``: where ``dump()`` proved xceadmin can be a local-file +decoder with no network at all, ``extres status`` reads the status JSON +``xceesd`` writes on its own host (``esd_status_write``, +``src/esd/src/esd_status.c``) and formats it for the operator. No wire, +no ``--controller``, no TLS -- the daemon's status file is deliberately +the observability surface until the PR-2 proxy formalizes a status +channel (design.md 3.6; a temporary remote protocol would be +redesigned out from under us by PR-2's provider-lease model). + +Split into its own file (rather than appended to ``test_cli.py``, 1187 +lines at the time of writing) to keep both files under the project's +800-line guideline -- the same split precedent as +``test_cli_scheduler_mock.py``. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from xceadmin import cli as cli_module +from xceadmin.cli import cli + + +# --------------------------------------------------------------------------- +# Fixtures: realistic status JSON in the exact shape esd_status_write() +# emits (field order aside -- we parse, we never substring-match). +# --------------------------------------------------------------------------- + +_NOW = 1_800_000_000 # a fixed "now" for deterministic age rendering + + +@pytest.fixture +def status_file(tmp_path: Path) -> Path: + """A status file exactly as esd_status_write() writes it for two + sources: one fresh, one stale (last_ok_time 3600s in the past of + _NOW against the daemon's 150s default limit). + """ + path = tmp_path / "esd-status.json" + payload = { + "daemon": "xceesd", + "pid": 12345, + "started_at": _NOW - 7200, + "generated_at": _NOW, + "pushes_ok": 142, + "pushes_fail": 3, + "sources": [ + { + "name": "Verilog_XL", + "last_value": 2.0, + "last_ok_time": _NOW - 5, + "stale": False, + }, + { + "name": "VCS", + "last_value": 7.0, + "last_ok_time": _NOW - 3600, + "stale": True, + }, + ], + } + path.write_text(json.dumps(payload)) + return path + + +def _freeze_now(monkeypatch: pytest.MonkeyPatch) -> None: + """Pin time.time() to _NOW so age rendering is deterministic.""" + monkeypatch.setattr(cli_module.time, "time", lambda: _NOW) + + +# --------------------------------------------------------------------------- +# Formatting helper: pure functions tested directly (the _format_record_ +# result precedent in test_cli.py). +# --------------------------------------------------------------------------- + + +def test_format_status_renders_daemon_identity_and_counters() -> None: + from xceadmin.cli import _format_extres_status + + payload = json.loads( + (Path(__file__).parent / "data" / "esd-status-sample.json").read_text() + ) if (Path(__file__).parent / "data" / "esd-status-sample.json").exists() else { + "daemon": "xceesd", + "pid": 12345, + "started_at": _NOW - 7200, + "generated_at": _NOW, + "pushes_ok": 142, + "pushes_fail": 3, + "sources": [], + } + + text = _format_extres_status(payload, now=_NOW) + + assert "xceesd" in text + assert "12345" in text + assert "142" in text + assert "3" in text + + +def test_format_status_marks_stale_source_with_age() -> None: + from xceadmin.cli import _format_extres_status + + payload = { + "daemon": "xceesd", + "pid": 1, + "started_at": _NOW - 100, + "generated_at": _NOW, + "pushes_ok": 0, + "pushes_fail": 0, + "sources": [ + { + "name": "VCS", + "last_value": 7.0, + "last_ok_time": _NOW - 3600, + "stale": True, + } + ], + } + + text = _format_extres_status(payload, now=_NOW) + + assert "VCS" in text + assert "STALE" in text + assert "3600s" in text + + +def test_format_status_fresh_source_shows_age_only() -> None: + from xceadmin.cli import _format_extres_status + + payload = { + "daemon": "xceesd", + "pid": 1, + "started_at": _NOW - 100, + "generated_at": _NOW, + "pushes_ok": 0, + "pushes_fail": 0, + "sources": [ + { + "name": "Verilog_XL", + "last_value": 2.0, + "last_ok_time": _NOW - 5, + "stale": False, + } + ], + } + + text = _format_extres_status(payload, now=_NOW) + + assert "Verilog_XL" in text + assert "STALE" not in text + assert "5s" in text + + +def test_format_status_source_with_no_value_yet() -> None: + from xceadmin.cli import _format_extres_status + + payload = { + "daemon": "xceesd", + "pid": 1, + "started_at": _NOW - 100, + "generated_at": _NOW, + "pushes_ok": 0, + "pushes_fail": 0, + "sources": [ + { + "name": "Verilog_XL", + "last_value": None, + "last_ok_time": 0, + "stale": False, + } + ], + } + + text = _format_extres_status(payload, now=_NOW) + + assert "Verilog_XL" in text + assert "no value yet" in text + + +# --------------------------------------------------------------------------- +# CLI wiring: arg parsing, file reading, error translation. +# --------------------------------------------------------------------------- + + +def test_cli_help_shows_extres_command() -> None: + runner = CliRunner() + result = runner.invoke(cli, ["--help"]) + + assert result.exit_code == 0 + assert "extres" in result.output + + +def test_extres_status_reads_local_file(status_file: Path, monkeypatch: pytest.MonkeyPatch) -> None: + _freeze_now(monkeypatch) + + runner = CliRunner() + result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)]) + + assert result.exit_code == 0, result.output + assert "xceesd" in result.output + assert "Verilog_XL" in result.output + assert "VCS" in result.output + + +def test_extres_status_defaults_to_etc_path(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """With no --status-file, the default path is the deployed default + (/var/run/xce/esd-status.json -- where esd.yaml's production + status_file default points); a missing default file is a clean + error, not a traceback. + """ + monkeypatch.setattr( + cli_module, "DEFAULT_STATUS_FILE", str(tmp_path / "nonexistent.json") + ) + + runner = CliRunner() + result = runner.invoke(cli, ["extres", "status"]) + + assert result.exit_code != 0 + assert "nonexistent.json" in result.output + + +def test_extres_status_missing_file_clean_error(tmp_path: Path) -> None: + runner = CliRunner() + result = runner.invoke( + cli, ["extres", "status", "--status-file", str(tmp_path / "gone.json")] + ) + + assert result.exit_code != 0 + assert "gone.json" in result.output + assert "Traceback" not in result.output + + +def test_extres_status_invalid_json_clean_error(tmp_path: Path) -> None: + bad = tmp_path / "bad.json" + bad.write_text("{not json") + + runner = CliRunner() + result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)]) + + assert result.exit_code != 0 + assert "Traceback" not in result.output + + +def test_extres_status_wrong_schema_clean_error(tmp_path: Path) -> None: + """JSON that parses but is not an esd status object (e.g. someone + pointed --status-file at some other JSON) fails loud and clean. + """ + bad = tmp_path / "other.json" + bad.write_text(json.dumps({"hello": "world"})) + + runner = CliRunner() + result = runner.invoke(cli, ["extres", "status", "--status-file", str(bad)]) + + assert result.exit_code != 0 + assert "Traceback" not in result.output + + +def test_extres_status_no_controller_needed(monkeypatch: pytest.MonkeyPatch, status_file: Path) -> None: + """The command must not touch the controller: XCE_CONTROLLER_PEERS + pointing at a dead endpoint changes nothing (the file is local). + """ + monkeypatch.setenv("XCE_CONTROLLER_PEERS", "127.0.0.1:1") + monkeypatch.setenv("XCEADMIN_CONTROLLER", "127.0.0.1:1") + + runner = CliRunner() + result = runner.invoke(cli, ["extres", "status", "--status-file", str(status_file)]) + + assert result.exit_code == 0, result.output diff --git a/src/xceadmin/tests/test_config_dump.py b/src/xceadmin/tests/test_config_dump.py index b2b66e55..3a94387f 100644 --- a/src/xceadmin/tests/test_config_dump.py +++ b/src/xceadmin/tests/test_config_dump.py @@ -49,6 +49,32 @@ def test_dump_lists_each_resource_with_decoded_type_and_order(full_cluster_bin): assert "consumable=False" in text +def test_dump_renders_builtin_and_external_flags(full_cluster_bin): + """The flag bits that change a resource's semantics must be visible in a + dump: builtin (worker/controller hardcode the values -- the repo fixture's + load indices and host attributes) and external (values come only from the + extres daemon's pushes -- the extlic pool shape). Both were silently + invisible before: every builtin resource dumped as a plain user resource, + and an external pool dumped as bare shared=true. + """ + text = build.dump_cluster_config(str(full_cluster_bin)) + mem_line = next(l for l in text.splitlines() if "name=mem type=numeric" in l) + assert "builtin=True" in mem_line + # a user-declared resource must not show a bare "builtin=" at all + assert "builtin=False" not in text + + import yaml as _yaml + configs = str(FIXTURES_DIR / "full" / "configs") + with open(configs + "/resources.yaml", encoding="utf-8") as fh: + names = [r["name"] for r in _yaml.safe_load(fh)["resources"] + if (r.get("resourcemap") or []) and + (r["resourcemap"][0].get("location") or []) == ["external"]] + assert names, "fixture must contain an external resource for this test" + for name in names: + line = next(l for l in text.splitlines() if f"name={name} " in l) + assert "external=True" in line, line + + def test_dump_renders_mxj_auto_detect_sentinel_not_raw_uint32(full_cluster_bin): """The exact bug that motivated this feature: hosts.yaml's `default` host has `max_slot: -1`, which wire-encodes as HOSTCONFIG_MXJ_AUTO_DETECT diff --git a/src/xceadmin/tests/test_mapping.py b/src/xceadmin/tests/test_mapping.py index 88ceb67c..089e64d0 100644 --- a/src/xceadmin/tests/test_mapping.py +++ b/src/xceadmin/tests/test_mapping.py @@ -176,6 +176,29 @@ def test_map_list_section_resources_populates_mapped_fields_only(cluster_config) assert r0.lastReport == 0 +def test_map_list_section_resources_builtin_flag(cluster_config): + """``builtin: true`` (resources.yaml) -> XCE_RESF_BUILTIN (0x01) OR'd + on top of the other bits: the load indices (mem/io/r1m...) and host + attributes (type/model/maxmem...) are produced by hardcoded worker/ + controller reporters, and the flag tells them apart from user-declared + resources (the CLI's BUILTIN column renders it). Absent/false must + stay 0 -- the same convention as consumable/release. + """ + items = [ + {"name": "mem", "type": "numeric", "interval": 60, + "consumable": True, "builtin": True, "description": "memory"}, + {"name": "mylic", "type": "numeric", "consumable": True, + "builtin": False, "description": "user pool"}, + {"name": "plain", "type": "numeric"}, # builtin absent + ] + mapping.map_list_section(_spec_for("resources"), items, cluster_config) + + r0, r1, r2 = cluster_config.resources + assert r0.flags == 0x80 | 0x01 # CONSUMABLE | BUILTIN + assert r1.flags == 0x80 # explicit false -> no BUILTIN bit + assert r2.flags == 0 # absent -> no BUILTIN bit + + def test_map_list_section_noop_on_empty_or_missing_items(cluster_config): mapping.map_list_section(_spec_for("resources"), [], cluster_config) mapping.map_list_section(_spec_for("resources"), None, cluster_config) @@ -248,6 +271,42 @@ def test_map_list_section_resourcemap_maps_distributions_and_all_flag(cluster_co assert r4.flags == 0 +def test_map_list_section_resourcemap_external_location_flag(cluster_config): + """resourcemap location==["external"] (external-license-resources): the + resource carries XCE_RESF_SHARED | XCE_RESF_GLOBAL | XCE_RESF_EXTERNAL + (0x08|0x04|0x10) -- cluster-wide pool, but its values come only from the + extres daemon's free-count pushes, never from a config distribution or + worker reports. A bare ["external"] (no value) also maps definition-only + -- the daemon owns the values entirely. + """ + items = [ + { + "name": "verilog_xl", + "type": "numeric", + "consumable": True, + "resourcemap": [{"location": ["external"]}], + }, + { + "name": "extlic_valued", + "type": "numeric", + "resourcemap": [{"location": ["external"], "value": 7}], + }, + ] + mapping.map_list_section(_spec_for("resources"), items, cluster_config) + + r0, r1 = cluster_config.resources + + # CONSUMABLE | SHARED | GLOBAL | EXTERNAL + assert r0.flags == 0x80 | 0x08 | 0x04 | 0x10 + assert len(r0.distributions) == 1 + assert [str(h) for h in r0.distributions[0].hosts] == ["external"] + + # valued variant: same flags, the distribution value still maps (the + # controller refuses to seed it -- EXTERNAL skips seed_sharedinst) + assert r1.flags == 0x08 | 0x04 | 0x10 + assert r1.distributions[0].value.u.fvalue == 7.0 + + def test_map_list_section_resourcemap_real_repo_configs(cluster_config): """The mapper handles the real src/configs/resources.yaml: every resourcemap entry lands as a distribution, and the two [all]+value @@ -271,6 +330,18 @@ def test_map_list_section_resourcemap_real_repo_configs(cluster_config): for name in ("staticres1", "staticres2", "staticres3", "host_quota"): assert by_name[name].flags & (0x08 | 0x04) == 0 + # builtin:true resources (the worker load indices + host attributes the + # repo config declares) carry XCE_RESF_BUILTIN; user-declared shapes + # (staticres*, extlic) do not. + for name in ("slots", "mem", "r1m", "r15m", "r15s", "ut", "pg", "ls", "it", + "swp", "io", "tmp", "type", "model", "cpuf", "maxcpus", + "maxmem", "maxswap", "maxtmp", "ndisks"): + assert by_name[name].flags & 0x01 == 0x01, f"{name} missing BUILTIN" + for name in ("dynres1", "dynres2", "staticres1", "staticres2", "staticres3", + "staticres4", "res1", "sres1", "bigmem", "strres1", "strres2", + "host_quota", "gstring", "extlic"): + assert by_name[name].flags & 0x01 == 0, f"{name} unexpectedly BUILTIN" + # --------------------------------------------------------------------------- # map_list_section: users.yaml/hosts.yaml -> ugroups/hgroups (named groups)