TMax v1.1

In TMax v1.0, we introduced a simple recipe for training terminal agents with RL: a synthetic generation pipeline for diverse terminal environemtns, and an outcome-only DPPO training method with then sota results.Hamish Ivison*, Junjie Oscar Yin*, Rulin Shao*, Teng Xiao, Nathan Lambert, and Hannaneh Hajishirzi. “TMax: A simple recipe for terminal agents.” arXiv preprint arXiv:2606.23321, 2026. *Co-first authors.

In v1.1, we improve TMax receipe with cleaner data and better training infrastructure.

1. Auditing and repairing TMax environment

TMax v1.0 deliberately skipped teacher validation during environments construction. The pipeline only checked that each environment builds, and relied on our RL training to drop tasks the model can never solve via active sampling and zero-std filtering (a group where every rollout scores zero carries no gradient). That handles tasks that are too hard, but it does not handle the correctness and difficulty of verifiers, which can be too easy or hard for the specific task.

So in TMax v1.1, we add a single audit-and-repair pass after data generation, inspired by River.Yao et al., “River,” 2026. River uses rubric-based repair and filtering on existing datasets; we apply the same idea to our own generated data.

On top of the existing pipeline, we add the following:

• Auditor. GPT-5.6 Sol reads each task’s instruction, environment setup, and verifier, and assigns one of eight labels, such as verifier too weak, answer leak, instruction–verifier mismatch, or clean.
• Repairer. For the three most common fixable labels, GPT-5.6 Sol gets the original task plus the auditor’s diagnosis, and writes a hardened verifier or a cleaned image specification.
• Rebuild and spot-check. We rebuild the repaired images, check that they still build with the dependencies they need, and drop everything else that was flagged.

Below we show the audits for TMax-v1.0:

The audits identify weak verifiers as the dominant weakness. For example, a weak verfier checks that an output file exists and matches one fixed expected value, but never runs the program the task asked for. An agent can hack by simply hardcoding the answer. Below are some example verdicts from the audit.

We also find that roughly 11% of tasks leak the answer into the environment. Most often, the setup leaves behind the reference output that the verifier compares against, like the expected CSV in the example above, so an agent can copy it instead of doing the task. The file can’t simply be deleted, since the verifier still needs it. In our early tests, the repairer often kept the file and made the verifier reject an exact copy instead. But that is easy to get around: agent can simply copy the file, append a comment, and the output passes. The repairer therefore rebuilds the image without the leaked file and moves it into the tests, which are copied into the sandbox only after the agent has finished.

As in v1.0, we still do not filter out tasks a strong model can’t solve and rely on RL’s zero-variance filtering.

We find this audit to repair step critical in improving data quality. Holding everything else fixed, training on the repaired data improves Terminal-Bench Lite by 5.8 points and Terminal-Bench 2.1 by 2.3 points over the unrepaired v1.0 data, despite v1.1 having ~1k fewer data points.

2. Results

Comparing TMax-v1.1-13k against 12 other RL datasets

Since v1.0 we have doubled the set of baselines, adding six recent datasets: LiteCoder, SETA, CalibForge, Terminal-Lego, RST, and Facet.For several of these datasets, this is the first time they have been used for RL training in the open.

We apply RL training to Qwen3.5-9b and our harness with new v1.1 data. We find that training on TMax-v1.1-13K exceeds training on all other dataets. Specifically, TMax-v1.1-13k has the highest mean on both benchmarks: 65.6 on TB Lite and 29.6 on TB 2.1.

Every model size improves under 30B

We apply the same TMax recipe to the other Qwen3.5 sizes and to Qwen3.6-27B.

On Terminal-Bench 2.0, TMax v1.1 improves over v1.0 at every size: 2B goes from 2.9 to 9.4, 4B from 18.9 to 25.1, 9B from 27.2 to 33.0, and 27B from 42.7 to 44.9.We re-ran the base models for v1.1 (three trials per task instead of five), so their scores differ slightly from the v1.0 post. For example, Qwen3.6-27B moved from 39.6% to 42.7%.

TMax-v1.1-9B now beats every prior terminal agent up to 30B parameters in our comparison, including CalibForge-30B-A3B (32.6%), as well as Claude Haiku 4.5 and GPT-5-mini. Although Qwen3.6-27B is already a strong terminal model, we still improve it by 2.2 points, and TMax-v1.1-27B outperforms the 1T-parameter Kimi-K2.5 (43.2%).

The v1.0 generalization results also hold for TMax-v1.1-9B. After RL training on our terminal tasks, our model improves SWE-bench Verified from 46.2 to 55.3 and AIME’24/25 (solved with the terminal) from 72.2 to 93.3, and it gains 8 to 14 points on TB Lite under every other harness we tried (OpenHands, mini-SWE-agent, Terminus-2).

Bootstrapping data without a strong teacher

v1.0 used a strong model, Gemini-3-Pro, to write every environment, and v1.1 adds GPT-5.6 Sol for auditing and repair. A natural objection is that our generation pipeline mainly relies on distillation. To test this, we reran the entire pipeline (generation, auditing, and repair) with Qwen3.5-9B, the same model we primarily trained on. Of the 5k environments it generated that built and passed their initial-state check, auditing and repair kept 3.1k.

The self-generated data improves Qwen3.5-9B by 12.6 points on TB Lite (49.4 to 62.0) and 6.8 points on TB 2.1 (20.5 to 27.3). That beats more than half of the prior datasets, despite being smaller than most of them, and lands within a few points of TMax-v1.1-13k. While the pipeline benefits from a strong teacher, it can achieve competitive results with bootstrapping alone.

We think the bootstrapping gains come down to the asymmetry between verification and solution. On average, building and verifying an environment is easier than solving it. The catch is that the 9B model writes broken environments more often. In our self-generation run, only about 1/6 of its environments built and passed their initial-state check, so it took roughly six times as many attempts per usable environment as with Gemini.

3. Stable RL Training on Terminal Agents

In v1.0, long training runs were often unstable, and some collapsed after a few hundred steps. Our original fixes, an FP32 LM head to reduce the mismatch between trainer and inference engine and larger groups, helped but did not completely remove the instability.

In v1.1, we find our original sandbox serving infrastructure to be easily overwhelmed. To save cost, our v1.0 relies on Podman sandboxes which ran on the same nodes as the inference engines. This means both the inference engine and sandbox serving services is competing for CPU and memory. Under load, sandboxes slowed down or failed, which added noise and errors to the rollouts.

This conflict turned out to be the main cause. Once we reduced infrastructure errors, RL training ran smoothly for 1,000 steps.

Why sandbox noise breaks RL

Every rollout is a long agent episode, around 25 commands, inside its own sandbox, and many layer underneath can fail. An image fails to pull, a node is preempted, a command hangs because the node is overloaded, or the rollout spends its time budget waiting for a free sandbox. In v1.0, rollouts with these issues all are assigned 0 reward.

This is a critical error for DPPO/GRPO. Group-based RL scores each rollout against its groups, so a “fake zero” rollouts can actively pushes the policy away from a what might be a good rollouts if not for hangs in sandboxes.

A dedicated sandbox fleet

We improve our RL infrastructure from two ends. First, we separate sandbox serving from inference serving for rollouts to avoid resource competition. We built our own system for this, Sandfleet.Fair warning: Sandfleet is a fairly vibe-coded tool. It works for our setup, but expect rough edges.

Second, we flag every rollout that fails because of the infrastructure, so the trainer can tell these apart from rollouts the model failed and leave them out of training.

• Separate nodes for sandboxes. Sandboxes run on CPU nodes that the controller requests from Slurm on demand and releases when idle, so they no longer compete with inference for CPU, memory, and disk. Each rollout holds one 2-CPU sandbox lease. Leases expire unless renewed, and lost or orphaned sandboxes are cleaned up, so a multi-day run does not slowly leak capacity.
• Prebuilt task images. We build and check one image per task before training starts, so nothing is pulled from a registry mid-run. A missing image is flagged as an error.
• Masking infrastructure failures. When a sandbox’s worker disappears or the sandbox stops running mid-episode, the rollout is flagged as an infrastructure failure. The trainer drops flagged rollouts and computes advantages from the rest of the group, skipping any group left with fewer than two valid rollouts. Failures the policy causes, like running out of memory, still score 0. If a sandbox can’t be set up at all, the run stops rather than train without it.
• Rollout time limits. A rollout’s time budget starts once it holds a sandbox, not while it waits in the queue. A command that times out comes back to the agent as an observation instead of ending the episode.

With the same recipe and data as v1.0, training now runs to 1,000 steps without collapsing. The same fleet also runs our TB Lite and TB 2.1 evaluations, at a fraction of the cost of a hosted sandbox service.These evaluations use a fixed subset (96 of 100 TB Lite tasks, 88 of 89 TB 2.1 tasks), skipping the few tasks that rootless Apptainer cannot run reliably. Final TB 2.0 numbers still use Daytona for sandboxes.

Citation

Ivison, Hamish and Yin, Junjie Oscar and Shao, Rulin and Xiao, Teng and
Lambert, Nathan and Hajishirzi, Hannaneh, "Tmax: A simple recipe for terminal
agents", arXiv preprint arXiv:2606.23321, 2026.

@misc{ivison2026tmaxsimplerecipeterminal,
      title={Tmax: A simple recipe for terminal agents},
      author={Hamish Ivison and Junjie Oscar Yin and Rulin Shao and Teng Xiao and Nathan Lambert and Hannaneh Hajishirzi},
      year={2026},
      eprint={2606.23321},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.23321},
}

← All posts