SWE-TaskFlow: SWE-Bench Pro

This dataset represents SWE-Bench Pro ran through our SWE-TaskFlow approach. SWE-TaskFlow transforms any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In our case, we transformed SWE-Bench Pro to fit our Production Sessions: 4,782 agent sessions of real software engineers in JetBrains IDEs. Thus, the benchmark preserves the original tests but makes the tasks harder and multi-step.

Data Paper Paper pre-print