---
id: "bug-563"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "lib"
severity: "unknown"
timestamp: "2026-06-15T09:16:53.679025Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/lib/request.c"]
tags: ["joblist", "workerlist", "jobsub", "ha", "check_controller", "c-running", "cluster_start", "orphaned-function", "xce_reply_get", "SOLVED"]
related: ["bug-562"]
legacy: true
---

# bug-563: jobsub returns ok (status 0, jobid) but joblist AND jobinfo show the job absent on a stable single…

## Description
jobsub returns ok (status 0, jobid) but joblist AND jobinfo show the job absent on a stable single primary (ctrl-1); reslist works.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
SOLVED (reproduced locally via /xce-build + 3-node deploy-ha, ASAN). cluster_start() — the ONLY place that sets c->running=1 — is ORPHANED (zero callers; grep confirms). The HA startup path (myapp_run) starts iomgr/vtaskmgr/delaymgr/persist_start itself but never calls cluster_start, so c->running stays 0 (calloc default) on the primary. check_controller() (the streaming gate passed to xce_reply_get for joblist/workerlist) returns -EINVAL when !c->running; xce_reply_get then sets nc=0 (chunk count), shipping an EMPTY list to the client even though collect_jobs/key_find found the job. reslist passes checkf=NULL so it was immune. DIAG proof: 'reply_get send: table=2 pos=0 nc=0 nrows=1 checkstatus=-22' (pre-fix) -> 'nc=1 ... checkstatus=1' (post-fix). Single-node/legacy cluster_init path called cluster_start, which is why non-replicated worked.

## Fix
Set c->running=1 at the TOP of cluster_run() (src/controller/cluster.c), symmetric with the existing c->running=0 on shutdown — the orphaned cluster_start() used to set it. c->running is a liveness/shutdown flag: check_controller() is polled PER CHUNK inside xce_reply_get()'s streaming loop so a long multi-chunk fetch (e.g. thousands of jobs) ABORTS mid-stream (nc=0) if a shutdown request arrives during it. The bug: running started 0 and was never set under HA, so the gate fired on every reply, not just during shutdown. Fix restores running=1 for the run-loop lifetime; the shutdown abort path (running=0 when cluster_run's cbuf_get loop breaks on the stop signal) is unchanged. Verified E2E on a local 3-node deploy-ha (ASAN): jobs now appear in joblist; DIAG showed checkstatus/nc flip -22/0 -> 1/1. Final fix is in cluster_run (not myapp_run — moved there so the 1<->0 lifecycle is co-located in the function that receives the shutdown signal).

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-562