diff --git a/.wolf/buglog.json b/.wolf/buglog.json index 890d9501..1694cb2d 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -63,7 +63,7 @@ "timestamp": "2026-10-27T03:27:47Z", "last_seen": "2026-10-27T03:27:47Z", "occurrences": 1, - "error_message": "cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).", + "error_message": "cluster_build_resources aborted at first EEXIST: new resource defs never registered\n\n**Observed:**\nController log on t1, after pushing a cluster.bin whose resources section ADDED\nthe new `extlic` definition (external-license-resources E2 Phase 1):\n\n```\n[ERROR] [log_update_resval_failure] update_resval `extlic': No such file or directory\n```\n\nThe xceesd daemon joined and pushed extlic values, but the controller had\nno `extlic` registration at all (update_resval key_find → -ENOENT).\n\n**Expected:** a config activation that adds a new resource definition must\nregister it in the running controller's resmgr, so daemon pushes land.\n\n**Impact:** every newly-added resource definition was silently inert until\nthe controller process was restarted (boot-time registration path did run).\nFor E2 this meant the whole extres-daemon chain was dead on a live controller.\n\n**Found by:** E2 Phase 1 live E2E on t1 (spec .kiro/specs/external-license-resources design.md §3.5 checklist).", "file": "src/controller/cluster.c, src/controller/testsuite/test_config_resource_reconcile.c", "root_cause": "`cluster_build_resources()` looped over the decoded config's resource list\ncalling `makeResource()` for each. `makeResource_` (src/lib/resobj.c:244-247)\nreturns NULL with errno=EEXIST when the name is already registered — which is\nthe COMMON case at activation (boot already registered some definitions). The\nold loop treated that rc as fatal and aborted the whole loop, so every\ndefinition AFTER the first already-registered one was never registered.", "fix": "Rewrite the loop on top of bug-1854's `resourceApplyDef()` (lookup+add under\none write lock, EEXIST-tolerant): RES_APPLY_ADDED/SAME proceed normally;\nRES_APPLY_CHANGED updates the registered object's definition fields\n(valtype/ordtype/flags/interval) IN PLACE — none of them touches the\nregistration slot (idx) or accumulated accounting; mirrors\nreconcile_config_queues' in-place refresh precedent (second live bug, see\nbug-1974 — kept here as one loop). Negative rc still propagates.\n\nVerification:\n- Test: `test_reconcile_resources_registers_new_defs_past_eexist` +\n `test_reconcile_resources_hot_applies_changed_definition`\n (src/controller/testsuite/test_config_resource_reconcile.c); RED before the\n fix (lic1 unregistered / flags stayed 0x00), GREEN after.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 E2E — config gen=15 push with extlic; xceesd push landed in\n sharedinst (extlic=4), rusage[extlic=3] job RUNNED; second job PENDed at\n free=1; release re-admitted it.", @@ -76,7 +76,9 @@ "related_bugs": [ "bug-1854" ], - "fix_commits": [], + "fix_commits": [ + "2fcacac2" + ], "record": ".wolf/bugs/bug-1973.md" }, { @@ -103,7 +105,9 @@ "bug-1854", "bug-1973" ], - "fix_commits": [], + "fix_commits": [ + "2fcacac2" + ], "record": ".wolf/bugs/bug-1974.md" }, { @@ -116,7 +120,7 @@ "timestamp": "2026-10-27T03:40:05Z", "last_seen": "2026-10-27T03:40:05Z", "occurrences": 1, - "error_message": "xceesd never reconnects after controller restart when its value is unchanged (debounce suppresses pushes)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).", + "error_message": "xceesd idle on dead connection after controller restart (stable value: no re-push)\n\n**Observed:**\nOld xceesd (pid 162609) sat idle on a dead connection for 50+ minutes after\nthe controller was restarted:\n\n```\n(no socket() attempts; strace showed only collector forks + epoll_wait;\n /proc/net socket table had no trace of the controller connection)\n```\n\nLast log line was the boot-time `esd full re-push: 0/1 source(s)` — no\nfurther activity.\n\n**Expected:** within one poll cycle of the controller's restart the daemon\nmust re-establish the connection and re-push its value (the new primary has\nno record of the daemon or its value).\n\n**Impact:** after any controller restart/failover with a stable license\nvalue, the controller restarts with extlic read 0 (seed skipped for EXTERNAL\nby design) and no daemon re-push — every rusage[extlic] job PENDs forever\nwith NO_RESOURCE until the license value happens to change.\n\n**Found by:** E2 Phase 1 live E2E on t1 (controller restart step).", "file": "src/esd/src/esd.c, src/lib/client.c, src/lib/client_internal.h", "root_cause": "Three mechanisms conspired:\n1. `esd_poll_round()` debounces unchanged values (`last_value == v` → skip\n push) — with a stable value it pushes NOTHING.\n2. A reconnect inside the client library only fires while INSIDE a push's\n failover call — no pushes, no reconnect.\n3. `esd_on_reconnect` (the rejoin hook) only runs on a successful rotate;\n with no push attempt it never runs.", "fix": "`esd_run()`'s main loop now checks `xcecli_is_connection_broken()`\n(bug-1760's pure predicate — the transport's RDHUP handler sets\nXCE_CTX_ERR asynchronously) and forces one `esd_push_all()` when it latches.\nThe write attempt drives the rotate/reconnect; `esd_on_reconnect` keeps the\nfull re-push contract for the failover path.\n\nVerification:\n- Test: covered by the live contract (client-side predicate already unit\n tested by bug-1760's suite); esd-side loop is integration-shaped (the\n trigger is a transport callback), so live E2E is the proof.\n- Suites: controller testsuite 1358 tests, 0 failures (build.dbg).\n- Live: t1 — controller killed and restarted; esd logged\n `connection broken -- forcing a full re-push to reconnect`, rotated to the\n new primary, re-pushed; `xcecli resources distribution list` shows extlic=4\n and a fresh rusage[extlic=3] job RUNS.", @@ -131,7 +135,9 @@ "bug-1973", "bug-1974" ], - "fix_commits": [], + "fix_commits": [ + "2fcacac2" + ], "record": ".wolf/bugs/bug-1975.md" } ] diff --git a/.wolf/bugs/bug-1973.md b/.wolf/bugs/bug-1973.md index 2d913360..80b34298 100644 --- a/.wolf/bugs/bug-1973.md +++ b/.wolf/bugs/bug-1973.md @@ -11,10 +11,11 @@ occurrences: 1 files: ["src/controller/cluster.c", "src/controller/testsuite/test_config_resource_reconcile.c"] tags: ["config", "reconcile", "E2", "external-license"] related: ["bug-1854"] -summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered on a running controller" +fix_commits: ["2fcacac2"] +summary: "cluster_build_resources aborted at first EEXIST: new resource defs never registered" --- -# bug-1973: cluster_build_resources aborted whole loop on first EEXIST so new resource definitions never registered on a running controller +# bug-1973: cluster_build_resources aborted at first EEXIST: new resource defs never registered ## Description **Observed:** diff --git a/.wolf/bugs/bug-1974.md b/.wolf/bugs/bug-1974.md index 706fa8a1..1b0eceb3 100644 --- a/.wolf/bugs/bug-1974.md +++ b/.wolf/bugs/bug-1974.md @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/controller/cluster.c", "src/controller/worker.c", "src/xceadmin/src/xcecommon/mapping.py"] tags: ["config", "flags", "E2", "external-license"] related: ["bug-1854", "bug-1973"] +fix_commits: ["2fcacac2"] --- # bug-1974: config activation kept stale resource definition (flags=0): daemon pushes refused with -EPERM diff --git a/.wolf/bugs/bug-1975.md b/.wolf/bugs/bug-1975.md index 03f18713..c4800e1c 100644 --- a/.wolf/bugs/bug-1975.md +++ b/.wolf/bugs/bug-1975.md @@ -11,6 +11,7 @@ occurrences: 1 files: ["src/esd/src/esd.c", "src/lib/client.c", "src/lib/client_internal.h"] tags: ["esd", "external-license", "reconnect", "E2"] related: ["bug-1760", "bug-1973", "bug-1974"] +fix_commits: ["2fcacac2"] summary: "xceesd idle on dead connection after controller restart (stable value: no re-push)" title: "xceesd idle on dead connection after controller restart (stable value: no re-push)" ---