A paper posted to arXiv on September 23 (arXiv:2609.28449) introduces SWE-Flux, a repository-level benchmark for dynamic execution reasoning in code. It contains 480 execution-grounded instances across 12 real Python repositories, with gold answers harvested automatically from instrumented test executions rather than written by hand or judged by LLMs. The benchmark covers single-test and multi-test questions over control flow, loops, program state, dataflow, exceptions, and program invariants. Evaluating five LLMs, the authors find the best model reaches only 37% accuracy: models do well on invariants, intra-procedural control flow, exceptions, and simple loops, but struggle with dataflow, inter-procedural execution, precise state reasoning, and suite-level aggregation.
[1][2]The paper answers a concrete question: LLMs can write code, but can they reason about what the code does when it runs? Existing repository-level QA benchmarks mostly test static understanding and often lean on LLM-based evaluation, while execution-reasoning benchmarks are limited to snippets or single functions. SWE-Flux differs on two axes — scale, moving execution reasoning to real repository level, and answer provenance, with gold answers produced automatically by instrumented test execution, bypassing the loop where an LLM grades itself. That design independently grounds runtime-behavior reasoning as a benchmark methodology.
The most informative part of the results (self-reported) is the strength/weakness split: models handle localized behavior (invariants, intra-procedural control flow, exceptions, simple loops) but collapse on dataflow, inter-procedural execution, precise state reasoning, and aggregating multiple test outcomes into a suite-level conclusion — the best model sits at just 37% overall. That split points to an actionable finding: LLM execution intuition is local, lacking the global reasoning needed to track variables flowing across procedures and to synthesize several test results into one conclusion. For real engineering, these are exactly the tasks where static-analysis tools and test frameworks are strong, so the fix may not be a stronger model but one that learns to call tools. The paper also contributes a byproduct: input perturbation automatically generates fresh benchmark variants, producing valid and substantially harder variants for almost 90% of selected instances — cheap benchmark growth that resists saturation.
Caveats: all results are self-reported; the benchmark covers Python only (12 repositories, 480 instances) with five evaluated models, and cross-language or cross-scale generalization is untested; gold answers depend on the correctness of instrumented execution, and coverage for complex repositories is not expanded. In the research line of letting models call execution tools, this paper sits alongside work that stops asking LLMs to guess runtime behavior from thin air: hand execution to a real interpreter and let the model reason and decide — SWE-Flux's contribution is laying the evaluation groundwork that division of labor needs.
One more observation worth making about the 37% ceiling: it is not a headline about model failure so much as a calibration of where current code LLMs actually stand. On tasks a human developer would call routine — what happens to this variable after this loop, does this exception propagate — the models are roughly usable; on the repository-level synthesis that senior engineers do, they are not. That gap is precisely what makes the benchmark useful for teams shipping coding agents: it tells them where to put verification hooks instead of trusting the model's self-report. And because the oracle-harvesting pipeline regenerates harder variants cheaply, the same methodology can track whether new models are genuinely improving at execution reasoning or just memorizing answer patterns. The authors leave cross-language coverage and larger repository corpora to future work, which is where the benchmark's real stress test will come from.
[1][2]