Meet SETA: Open Supply Coaching Reinforcement Studying Environments for Terminal Brokers with 400 Duties and CAMEL Toolkit

What does an finish to finish stack for terminal brokers seem like once you mix structured toolkits, artificial RL environments, and benchmark aligned analysis? A group of researchers from CAMEL AI, Eigent AI and different collaborators have launched SETA, a toolkit and setting stack that focuses on reinforcement studying for terminal brokers. The venture targets brokers that function inside a Unix model shell and should full verifiable duties underneath a benchmark harness resembling Terminal Bench.

Three major contributions:

• A state-of-the-art terminal agent on Terminal Bench: They obtain state-of-the-art efficiency with a Claude Sonnet 4.5 based mostly agent on Terminal Bench 2.0 and with a GPT 4.1 based mostly agent on Terminal Bench 1.0. The comparability is restricted to brokers that use the identical base mannequin.
• Scalable RL coaching with artificial terminal environments: The analysis group launch an preliminary artificial dataset with 400 terminal duties that cowl a variety of problem ranges. Out of those, 260 duties are used for RLVR finetuning of a Qwen3-8B mannequin.
• A clear agent design that generalizes throughout coaching and analysis frameworks: The identical agent implementation is used for each native process runs and the official Terminal Bench analysis harness.

Terminal Toolkit and log construction

The SETA code repository showcases a Terminal Toolkit that turns a language mannequin into an executable terminal agent. For every process run, the framework creates a structured log listing underneath analysis/terminal_bench_run. The README web page exhibits a concrete structure for a process referred to as play-zork.

Key information embrace:

• chatagent.log which data the total historical past of agent messages and gear calls together with take a look at outcomes.
• A periods listing with session_logs that seize terminal interactions from the toolkit.
• Inside session_logs, information resembling blocking_commands.log, session_run_zork_1_correct_path.log, session_zork-1.log, and session_zork_start.log retailer command output for various periods and modes.
• checks.log and checks.log.strip which report the take a look at run output, with the latter eradicating terminal management characters.

This construction provides a concrete technique to debug an agent. You may hint from excessive degree chat selections in chatagent.log all the way down to particular person shell instructions within the session logs and ensure success or failure from the take a look at logs.

For official Terminal Bench analysis, the GitHub repository supplies a separate entry level underneath analysis/terminal_bench_eval. A developer strikes into that listing and runs run_eval.sh for Terminal Bench 1.0 and run_tb2.sh for Terminal Bench 2.0.

Outcomes are written into analysis/terminal_bench_eval/run/{run_id}/outcomes.json. Job particular session logs are positioned underneath analysis/terminal_bench_eval/logs/camel_logs/{task_id}. The agent class that binds the CAMEL agent to the benchmark is applied in tbench_camel_agent.py.

Notice Taking Toolkit as persistent reminiscence

The analysis group additionally introduces a Notice Taking Toolkit described as persistent reminiscence for lengthy horizon duties. They present instance observe taking software calls the place the agent writes and reads notes in a structured manner whereas fixing terminal duties. The present public materials focuses on the existence of this toolkit and the examples of use. It doesn’t but describe a full coaching goal for observe utilization.

The vital level is that the agent has an specific channel the place it could externalize intermediate outcomes and hints, separate from the uncooked terminal buffer.

Understanding the efficiency

SETA’s agent harness achieves main outcomes on Terminal Bench. With Claude Sonnet-4.5 because the spine, the CAMEL terminal agent reaches 46.5% accuracy on Terminal Bench 2.0 throughout 89 actual world duties, rating first and outperforming the second system by 3 proportion factors, with particularly sturdy ends in git workflows, DevOps automation, and code safety duties. On Terminal Bench 1.0, a GPT 4.1 based mostly agent attains 35% accuracy, which is 4.7 proportion factors above the subsequent entry, once more inside the identical mannequin household. Compared, a supervised Qwen3 8B baseline attains 3.4% on Terminal Bench 2.0, and the Qwen3 8B terminal agent educated with the SETA RL pipeline improves over this baseline on the curated artificial environments.

Key Takeaways

• SETA is a joint group venture that gives each agent toolkits and artificial RL environments particularly for terminal brokers, aligned with the Terminal Bench analysis format.
• The framework studies state-of-the-art efficiency for CAMEL terminal brokers on Terminal Bench 1.0 and a couple of.0 when utilizing Claude Sonnet 4.5 and GPT 4.1 as the bottom fashions, evaluated towards brokers constructed on the identical mannequin households.
• The SETA RL dataset on Hugging Face comprises 400 artificial terminal duties, every packaged as process.yaml, Dockerfile, and run-tests.sh, with 260 duties used for RLVR finetuning of a Qwen3-8B based mostly agent.
• The open supply SETA codebase exposes a Terminal Toolkit with structured logging and a Notice Taking Toolkit for lengthy horizon reminiscence, and integrates straight with Terminal Bench analysis scripts and logging paths within the seta GitHub repository.
• The general design demonstrates a clear path from artificial RL environments to benchmark verified brokers, giving builders a reproducible stack to coach, debug, and consider terminal brokers somewhat than counting on advert hoc software calling examples.

Try the Blog, Technical details, GitHub Repo and Weights. Additionally, be happy to comply with us on Twitter and don’t overlook to affix our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Try our newest launch of ai2025.dev, a 2025-focused analytics platform that turns mannequin launches, benchmarks, and ecosystem exercise right into a structured dataset you’ll be able to filter, examine, and export.