---
id: "bug-1680"
status: "fixed"
status_updated: "2026-09-24"
status_note: ""
component: "apiserver"
severity: "data-loss"
timestamp: "2026-08-25T03:50:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/frontend/xcecli/filters/job_filter.go:58-60", "src/frontend/apiserver/utils/types.go:262-322", "src/frontend/apiserver/store/query.go:70-73"]
tags: ["xcecli", "apiserver", "job_filter.go", "types.go", "query.go", "job-status-filter", "cli-bug", "discovered-via-bug-1679"]
related: ["bug-1679"]
legacy: true
---

# bug-1680: `xcecli jobs list -a` ("list run,done,pending,suspended jobs" -- i.e

## Description
`xcecli jobs list -a` ("list run,done,pending,suspended jobs" -- i.e. show every job regardless of status) returns "no job found" even when jobs genuinely exist and are correctly stored with a real status. Discovered as a side effect of live-verifying bug-1679: two jobs that correctly showed PEND->RUN->DONE via the default query and via `-d` (done-only) both come back empty under `-a` specifically. Not a data-loss bug (the jobs ARE correctly stored and queryable through other flags -- `-d`, the default active-jobs view -- this is specifically `-a`'s own query construction that's broken) but a real, reproducible CLI defect: `-a` is documented as the way to see literally everything, and it shows nothing.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
xcecli's `-a` flag (list.go:347, `AllJobInfoFlag`) makes HandleJobFilter (job_filter.go:57-60) build the filter string `{job_status:notnull:}` -- a `notnull` operator on `job_status` with an EMPTY value token (no argument needed for null-checks, by design) -- instead of the normal `{job_status:in:(...)}`/`{job_status:notin:(...)}` shape every other status flag (-p/-r/-d/-s) produces. Server-side, `job_status`/`host_status` get SPECIAL bitmask handling in buildIntCondition (types.go:262-322), but that special-casing only triggers for the `in`/`notin`/`between` operators (the `if filter.Field == "job_status" || filter.Field == "host_status"` branch sits inside the `if filterOperator == "in" || filterOperator == "notin" || filterOperator == "between"` block, types.go:263-270) -- `notnull` falls through to the generic scalar path (types.go:316-322), which builds a plain `Condition{Column: "job_status", Operator: OpNOTNULL, Value: {TypeInt, 0}}` with no bitmask awareness at all. In isolation, store/query.go's own MatchesCondition evaluator (query.go:70-73) implements OpNOTNULL correctly (`fieldVal != nil`), which should trivially match every job (job_status is never actually absent/nil in a stored row) -- meaning the actual break is somewhere between condition construction and evaluation for this specific field/operator combination that was not fully traced (a debugger/print-tracing session would be needed to pin the exact line): plausible candidates are (a) the query engine routing job_status conditions through a bitmask-specific code path unconditionally regardless of operator, silently misinterpreting a non-bitmask NOTNULL Condition as a bitmask EQ/NE against value 0 (which would exclude every real job_status value, all of which are nonzero bitmask flags), or (b) some other job_status-specific special-casing elsewhere in the query pipeline that only handles the `in`/`notin`/`eq`/`ne` shapes types.go's own bitmask branch produces and never anticipated a bare NOTNULL condition on this field at all.

## Fix
FIXED (2026-08-25). Root cause fully pinned down (the prior entry's root_cause left the construction-to-evaluation gap untraced -- now traced exactly): src/frontend/apiserver/store/query.go's FilterRows() (the actual filter-evaluation loop, not something the prior investigation reached) routes ANY condition on job_status/host_status through the bitmask path purely based on IsBitmaskField(cond.Column) -- unconditional on the field NAME alone, never checking the condition's actual Operator. MatchesBitmaskCondition()'s own switch only implements OpEQ/OpNE (the only two shapes buildIntCondition's in/notin bitmask branch ever produces) and silently `return false` for any other operator -- so `{job_status:notnull:}` (Operator: OpNOTNULL, Value: {TypeInt, 0}, per types.go's null/notnull scalar path) gets routed into MatchesBitmaskCondition anyway, matches neither switch case, and returns false for every single row unconditionally. That's the entire bug: field-name-based routing that ignores which operator shape is actually on the condition.

Fix: FilterRows's routing condition now requires BOTH IsBitmaskField(cond.Column) AND cond.Operator being OpEQ or OpNE before taking the bitmask path; any other operator (OpNOTNULL, OpISNULL, or anything else that might ever target job_status/host_status) correctly falls through to the generic MatchesCondition evaluator, which already implements OpNOTNULL/OpISNULL correctly (fieldVal != nil / == nil) -- confirmed by the fix requiring zero changes to MatchesCondition or MatchesBitmaskCondition themselves, only the dispatch condition in FilterRows.

New src/frontend/apiserver/store/query_test.go (no test file existed for this package before -- confirmed via search, matches the prior entry's note): TestFilterRows_JobStatusNotNull_MatchesRealRows reproduces the exact bug (RED before the fix: 0 rows matched instead of 3; independently confirmed) and TestFilterRows_JobStatusBitmaskEQ_StillWorks locks in that the actual bitmask shape (OpEQ/OpNE, what every -p/-r/-d/-s status flag produces) is unaffected -- both pass after the fix. Full apiserver/store, apiserver/controllers, apiserver/models (CGO_ENABLED=0) and apiserver/backend + apiserver/backend/auth (full CGO via make test-backend) suites all pass, zero regressions.

Live-verified: rebuilt and hot-swapped the apiserver binary on a running cluster with 3 pre-existing DONE jobs (previously invisible to `-a`) plus one freshly-submitted job -- `xcecli jobs list -a` now correctly shows all 4 jobs across every status (3x DONE, 1x RUN), immediately, no restart needed.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1679