microsoft's open computer-use agent · Fara1.5-9B-FP8 · run entirely on our own hardware · 2026-07-28 · MIT license · $0 cloud
Fara is a vision-only browser agent: it looks at a screenshot of the page — no DOM, no
accessibility tree — thinks in text, and answers with one concrete action (click these pixels, type this,
scroll, visit this URL). Our harness executes that action in a real browser, takes a fresh screenshot, and
loops until the model says terminate with a final answer. The model runs on one GPU in our rack;
the browser can run anywhere — including attached to your own logged-in session.
The model server and the browser are decoupled — an OpenAI-compatible HTTP seam connects them. Screenshots go up, actions come back.
flowchart TD
subgraph ORCH["THE ORCHESTRATOR — any box"]
CLI["fara-cli harness
microsoft/fara + our patches"]
PW["Playwright controller
executes actions, takes screenshots"]
end
subgraph BROWSERS["THE BROWSER — pick one"]
SB["Sandbox chromium
fresh profile, no auth"]
CDP["Your own logged-in browser
CDP attach :9222"]
end
subgraph GPU["THE GPU BOX — one card"]
VLLM["vLLM server
Fara1.5-9B-FP8 · ~17GB slice"]
end
CLI --> PW
PW -->|"drives"| SB
PW -.->|"--cdp_url"| CDP
CLI -->|"screenshot + history"| VLLM
VLLM -->|"next action"| CLI
classDef gold stroke:#d4a73a,stroke-width:2px
class CDP,VLLM gold
sequenceDiagram
participant U as You
participant H as Harness
participant B as Browser
participant F as Fara 9B (GPU)
U->>H: task — "find X on this site"
H->>B: screenshot (1440x900 PNG)
B-->>H: pixels
H->>F: goal + last 3 screenshots + action history
Note over F: thinks in text, then emits ONE action
F-->>H: chain-of-thought + tool_call
H->>B: playwright executes click / type / scroll
Note over H,B: loop repeats ~3s per step until done
F-->>H: terminate (final answer)
H-->>U: answer + full trajectory + screenshots
Note over F: critical point? sign-in, payment, submit → PAUSES and asks the human first
The harness captures the viewport at 1440×900 — the resolution Fara was trained on. Only the last 3 screenshots ride along; older ones drop out of context.
Reasoning first, then exactly one grounded action:
The top story is the first
row under the site header...
<tool_call>{"name": "computer_use",
"arguments": {"action": "left_click",
"x": 342, "y": 187}}</tool_call>
The harness parses the XML, clicks those exact pixels in the real browser, waits for the page to settle, and the loop begins again with a fresh screenshot.
The harness spawns a fresh headless chromium — empty profile, no cookies, no auth. This is what the smoke test used. Can't touch your accounts even if it wanted to.
fara-cli --task "..." \ --endpoint_config endpoint_configs/vllm_config.json
Attach to your already-logged-in browser over the DevTools protocol. No stored credentials, no automation login to get flagged, and you watch every click live.
# 1 — relaunch your browser with a debug port brave --remote-debugging-port=9222 # 2 — point Fara at it fara-cli --task "..." --cdp_url http://localhost:9222 \ --endpoint_config endpoint_configs/vllm_config.json
| Action | What it does | Group |
|---|---|---|
left_click right_click double_click triple_click | Mouse clicks at predicted (x, y) pixel coordinates | mouse |
mouse_move left_click_drag | Cursor positioning and drag operations | mouse |
type key | Keyboard input — text entry and key combos | keyboard |
scroll hscroll | Vertical / horizontal page scrolling | navigate |
visit_url history_back web_search | Direct navigation and search | navigate |
pause_and_memorize_fact | Pins a fact so it survives even when old screenshots fall out of the windowused in our smoke run to hold the headline it read | memory |
ask_user_question | Stops and surfaces a question to the human | human gate |
wait terminate | Sleep N seconds / end the task with the final answer | control |
| Task type | Verdict | Why |
|---|---|---|
| Read / navigate / report"list the users", "check this setting", "find the price" | good today | WebVoyager-tier work — the 9B scores 86.6% here. Our smoke run was this class: flawless. |
| Multi-step writes on dense admin UIswizards, multi-page config changes | watch it | Harder-benchmark tier (63.4% Online-Mind2Web). Misclicks compound. Run headful, human watching. Fara's critical-points training makes it pause before sign-ins, payments, and submits — but a watcher still catches the fumbles. |
| Anything with an APIofficial SDKs, CLIs, admin APIs… | use the API | Deterministic beats vision-clicking. Fara's lane is the long tail of UIs with no API. |
| High-stakes records (legal / medical) | not this | The model card marks legal/high-stakes domains out of scope — and that matches our own hard rules for records that matter. |