Legato vs OSWorld's Qwen agent: which harness should you run an open model on?

On Qwen3.8-27B, Legato and OSWorld's QwenAgent scaffold took about the same time per task, and Legato solved more tasks: 71 of 96 rollouts against 53 of 96 over two runs on independent task sets. Use QwenAgent for OSWorld evaluations; use Legato to run an open model on your own sandboxes with bounded model calls.

Run 1: 36/48 vs 26/48. Run 2, on fresh data: 35/48 vs 27/48, with agent time 0.953 [0.798, 1.134] of QwenAgent's. Run 1 also looked faster, but that gap came from four runaway 32,768-token QwenAgent replies and did not replicate. Office tasks, not OSWorld; QwenAgent's model calls went through Legato's HTTP client (benchmarks).

	OSWorld QwenAgent (`mm_agents/qwen`)	Legato (`legato.Harness`)
Models	Any OpenAI-compatible endpoint; prompts and XML tool-call parsing written for Qwen	Any OpenAI-compatible chat API that accepts images
Desktops	OSWorld VMs (AWS, Azure, Docker, VMware, VirtualBox); actions are pyautogui code run in the VM	E2B Desktop, or any sandbox given a screenshot function and an execute function
Wait after action	0 s, the OSWorld runner's default for this agent	Until the screen stops changing, capped at 2 s per action
Screenshots kept	20, folded 10 at a time	8, dropped 4 at a time
Actions per turn	Every tool call in the reply	Up to 3, with one screenshot after them
Call bounds	32,768 output tokens; 130 s timeout; up to 5 attempts; 50 steps in the runner	4,096 output tokens with a cut-off note; 60 s timeout; 1 retry; 30 turns by default
License	Apache-2.0	Apache-2.0

When to use QwenAgent instead

OSWorld evaluation. It is the scaffold OSWorld ships for Qwen models. Use it when your numbers must be comparable with other runs of it.
Its own features: expanded actions, DashScope enable_thinking handling, an infeasible-task fallback, and OSWorld's VM providers with no adapter to write.
Long reasoning. It allows 32,768-token replies; Legato cuts at 4,096 unless you raise max_tokens.

Code

# pip install "legatolabs[e2b]"
import os
import legato
from e2b_desktop import Sandbox

agent = legato.Harness(model="qwen/qwen3.8-27b", base_url="https://openrouter.ai/api/v1",
                       api_key=os.environ["OPENROUTER_API_KEY"])   # or your vLLM server's /v1 URL
# non-Claude models get 0 to 1000 coordinates by default (coords="norm1000"), the Qwen-VL convention
desk = Sandbox.create()                                          # needs E2B_API_KEY
try:
    result = agent.run("Add a Total row under the table and save", legato.attach(desk))
    print(result.report())
finally:
    desk.kill()

Sources

xlang-ai/OSWorld, benchmarked at commit 8903ebe
mm_agents/qwen/README.md
main.py#L27-L36, #L44, #L53-L54, #L81
history.py#L7-L12
actions.py#L169-L170
client.py#L9, #L69-L78
run_multienv_qwen.py#L47-L48, #L72
desktop_env.py#L491-L530
README.md, LICENSE
Legato: harness.py, env.py, settle.py, rl/policy.py, rl/collect.py
Benchmarks

Edit this page on GitHub · Markdown