legato

Legato

Legato is an open-source computer-use harness for any OpenAI-compatible model (Qwen, DeepSeek, Claude, or your own vLLM or SGLang server, directly or through OpenRouter). It drives a desktop sandbox (E2B, or any sandbox you can screenshot and send input to) with a Claude-style computer tool. It waits until the screen is ready instead of sleeping a fixed time, and it bounds every model call. Apache-2.0.

pip install "legatolabs[e2b]"      # the package is legatolabs on PyPI; you import it as `legato`

Results

Same model, same desktops, same tasks, all arms run at the same time. The tasks are 24 Excel-to-PowerPoint office workflows on real Ubuntu desktops (LibreOffice). Times are agent time: first screenshot to last model turn. Ranges are 95% task-bootstrap intervals. Full method, raw data and losses: docs/benchmarks.md.

Model Legato compared with Time Success
Claude Sonnet 5.5 Anthropic's computer-use reference loop (re-implemented) 1.48x faster (0.678 [0.604, 0.757]) 48/48 vs 48/48
Qwen3.8-27B OSWorld's Qwen agent scaffold (mm_agents/qwen) about the same (0.953 [0.798, 1.134]) 71/96 vs 53/96 (74% vs 55%, two runs)

Quickstart

An open model through OpenRouter, on an E2B desktop:

import os
import legato
from e2b_desktop import Sandbox

agent = legato.Harness(model="qwen/qwen3.8-27b", base_url="https://openrouter.ai/api/v1",
                       api_key=os.environ["OPENROUTER_API_KEY"])
desk = Sandbox.create()                      # needs E2B_API_KEY
try:
    result = agent.run("Open the terminal and print the date", legato.attach(desk), max_steps=30)
    print(result.report())                   # where the time went: model, actions, waiting for the screen
finally:
    desk.kill()

The same harness works with Claude: legato.Harness(model="anthropic/claude-sonnet-5.5", ...). It switches to pixel coordinates and Anthropic's recommended screenshot size. It also works with your own server: base_url="http://localhost:8000/v1". From the command line:

legato run "Open the terminal and print the date" --model qwen/qwen3.8-27b --base-url https://openrouter.ai/api/v1

Any sandbox, without E2B: pass a screenshot function and an action executor.

env = legato.attach(screenshot=lambda: png_bytes(), execute=lambda action: send_input(action))

action is a legato.Action (type click / double_click / right_click / type / key / scroll / drag / wait, with pixel x, y, text, keys). More in docs/sandboxes.md and docs/models.md.

What it does differently

Legato Typical reference loop
After each action waits until the screen stops changing; on E2B the check runs inside the sandbox in one call fixed sleep (Anthropic's demo: 2 s; OSWorld runners: 0–5 s)
Actions per turn up to 3, one screenshot after them one per call, or all returned with a screenshot after each
Screenshot history old screenshots dropped 4 at a time past 8, so the provider's prompt cache keeps hitting all kept, or one dropped per step
Model calls timeout, retry, 4,096-token budget; a reply cut off before acting gets a note on the next turn often unbounded
Timing result.report() splits each run into model, action, waiting and screenshot time none

legato.Harness.reference(...) is a re-implementation of Anthropic's reference loop on the same plumbing, so you can measure the difference on your own tasks.

Reproduce the benchmarks

pip install "legatolabs[office,video]"
legato bench office generate --out bench --repeat 2          # 24 tasks: .xlsx data, .pptx decks, grader specs
legato rl collect --tasks bench/tasks.json --harness legato --policy-url https://openrouter.ai/api/v1 \
    --model qwen/qwen3.8-27b --api-key-env OPENROUTER_API_KEY --eval-only --osworld ~/OSWorld \
    --version legato --out runs/legato --desktops 6 --group 2 --max-steps 25
legato rl collect ... --harness qwen --no-ready-signal --fixed-wait 0 --version qwen-osworld --out runs/qwen-osworld
legato bench h2h --tasks bench/tasks.json --arm qwen-osworld=runs/qwen-osworld --arm legato=runs/legato \
    --ref qwen-osworld --out report

The desktops are OSWorld's Docker VMs, so you need a Linux host with KVM and an OSWorld checkout. See docs/benchmarks.md.

Comparisons and migration

MCP server

legato mcp gives Claude Code, Cursor or any MCP client a cloud desktop. Each action returns once the screen has settled, with the new screenshot. See docs/mcp.md.

pip install "legatolabs[e2b,mcp]"
claude mcp add legato -e E2B_API_KEY=your_key -- legato mcp

RL rollouts

legato rl collects graded desktop rollouts for computer-use RL. It records token logprobs and token ids, runs GRPO-style groups with oversampling and straggler cancel, caps in-flight requests, and profiles each batch. You bring the model server and the trainer. See docs/rl.md.

FAQ

More in docs/faq.md. License: Apache-2.0.

Edit this page on GitHub · Markdown