The AI Desktop Worker: GUI Applications Without APIs

The API gap

The modern enterprise runs on a patchwork of software. Some of it has APIs. Most of it doesn't. The CRM has a REST API. The ERP has a SOAP endpoint from 2008 that nobody maintains. The government compliance portal has no API at all — just a login page and a series of forms. The insurance claims system is a thick-client Java application that only runs on a desktop.

When you need to move data between these systems, the integration is a person. Someone logs into System A, copies a number, switches to System B, pastes it, fills in three fields, clicks Submit, goes back to System A, marks it as done. Repeat 200 times. Every day.

Traditional RPA (UiPath, Automation Anywhere, Blue Prism) tries to solve this with recorded click sequences. You record a macro, deploy it to a virtual machine, and pray that the UI doesn't change. When it does — and it always does — the macro breaks, the queue stalls, and an RPA engineer spends a day rebuilding the selector chain.

AI agents solve this differently. They don't replay recorded clicks. They see the screen, understand what's on it, and decide what to click. When a button moves from the left side to the right side, the agent finds it in its new location. When a dialog box pops up unexpectedly, the agent reads it and responds appropriately. The agent uses the application the same way a human does: by looking at it.

What a desktop container gives you

A desktop container is a cloud container running a full Linux desktop environment. When you create one via POST /api/desktops, you get:

• XFCE or KDE desktop — a complete graphical environment with window manager, taskbar, file manager, and system tray
• VNC access — connect via any VNC client or through the browser-based noVNC interface at the access_url
• Pre-installed software — LibreOffice (Calc, Writer, Draw, Impress), file manager, text editor, terminal emulator, PDF viewer, image viewer
• Root access — install any additional software with apt-get
• Network access — the desktop can reach the internet, internal services, and other containers
• Session token — secure access via the platform's proxy with time-limited tokens

The desktop container is the same environment a human would RDP or VNC into. The difference: instead of a human sitting at the screen, an AI agent controls it.

# Create a desktop container
export SANDBOX_API_KEY="ab0t_sk_live_YOUR_KEY_HERE"
export SANDBOX_URL="https://sandbox.dev.ab0t.com"

DESKTOP=$(curl -s -X POST "$SANDBOX_URL/api/desktops" \
  -H "Authorization: Bearer $SANDBOX_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "desktop_type": "ubuntu-xfce",
    "idle_timeout_minutes": 30,
    "metadata": {"task": "spreadsheet-processing"}
  }')

echo "Container: $(echo $DESKTOP | jq -r '.container_id')"
echo "Status:    $(echo $DESKTOP | jq -r '.status')"

# Poll until assigned, then open the desktop in your browser
CONTAINER_ID=$(echo "$DESKTOP" | jq -r '.container_id')

while true; do
  STATUS=$(curl -s "$SANDBOX_URL/api/containers/$CONTAINER_ID/status" \
    -H "Authorization: Bearer $SANDBOX_API_KEY")
  STATE=$(echo "$STATUS" | jq -r '.state')
  echo "State: $STATE"
  [ "$STATE" = "assigned" ] && break
  sleep 3
done

ACCESS_URL=$(echo "$STATUS" | jq -r '.access_url')
echo "Open in browser: $ACCESS_URL"

Open that access_url in your browser and you'll see a full Linux desktop rendered via noVNC. You can interact with it manually to test, or hand control over to an AI agent.

Available desktop types

How the agent controls the desktop

The AI agent controls the desktop using a screenshot-and-click loop. It takes a screenshot of the current screen, sends it to a vision model (Claude Sonnet 4.6 or Claude Opus 4.6), receives instructions about what to do next, and executes those instructions via VNC or xdotool commands. This is Claude's computer use capability applied to a dedicated container.

The control loop

import anthropic
import base64
import subprocess
import time

client = anthropic.Anthropic()

def screenshot() -> bytes:
    """Capture the current screen via xdotool/scrot."""
    subprocess.run(["scrot", "/tmp/screen.png", "-o"], check=True)
    with open("/tmp/screen.png", "rb") as f:
        return f.read()


def click(x: int, y: int):
    subprocess.run(["xdotool", "mousemove", str(x), str(y), "click", "1"])


def double_click(x: int, y: int):
    subprocess.run(["xdotool", "mousemove", str(x), str(y),
                    "click", "--repeat", "2", "--delay", "100", "1"])


def type_text(text: str):
    subprocess.run(["xdotool", "type", "--clearmodifiers", "--delay", "30", text])


def key_press(keys: str):
    """Press key combo like 'ctrl+s' or 'Return'."""
    subprocess.run(["xdotool", "key", "--clearmodifiers", keys])


def run_desktop_task(task_description: str, max_steps=50):
    """Run a task on the desktop using vision-guided interaction."""
    history = []

    for step in range(max_steps):
        # 1. Screenshot
        img = screenshot()
        b64 = base64.b64encode(img).decode()

        # 2. Ask Claude what to do
        messages = [{
            "role": "user",
            "content": [
                {"type": "text", "text": f"""You are controlling a Linux desktop to complete this task:

{task_description}

Previous actions taken: {json.dumps(history[-5:])}

Look at the screenshot and decide the next action.
Reply with exactly ONE line in one of these formats:
  CLICK x y
  DOUBLE_CLICK x y
  TYPE text to type
  KEY combo (e.g., KEY ctrl+s, KEY Return, KEY alt+F4)
  SCROLL up/down
  WAIT seconds
  DONE
  FAIL reason"""},
                {"type": "image", "source": {
                    "type": "base64", "media_type": "image/png", "data": b64}},
            ],
        }]

        response = client.messages.create(
            model="claude-sonnet-4-6-20250514",
            max_tokens=256,
            messages=messages,
        )
        action = response.content[0].text.strip()
        print(f"Step {step+1}: {action}")
        history.append(action)

        # 3. Execute
        if action.startswith("CLICK"):
            _, x, y = action.split()
            click(int(x), int(y))
        elif action.startswith("DOUBLE_CLICK"):
            _, x, y = action.split()
            double_click(int(x), int(y))
        elif action.startswith("TYPE"):
            type_text(action[5:])
        elif action.startswith("KEY"):
            key_press(action[4:])
        elif action.startswith("SCROLL"):
            direction = "5" if "down" in action else "4"
            subprocess.run(["xdotool", "click", direction])
        elif action.startswith("WAIT"):
            time.sleep(int(action.split()[1]))
        elif action == "DONE":
            print("Task complete.")
            return True
        elif action.startswith("FAIL"):
            print(f"Task failed: {action}")
            return False

        time.sleep(1)  # Let the UI update

    print("Max steps reached.")
    return False

Five use cases where desktop containers shine

1. Spreadsheet processing with LibreOffice

Your finance team receives a monthly spreadsheet from a partner. It needs to be reformatted: columns reordered, formulas updated, a pivot table created, the result exported as a PDF. There's no API for this — it's a spreadsheet manipulation task.

# Task: Process a financial spreadsheet in LibreOffice Calc
run_desktop_task("""
1. Open the file /workspace/monthly_report.xlsx in LibreOffice Calc
2. Delete columns E through G (they contain internal notes)
3. Sort the remaining data by column C (Amount) in descending order
4. Add a SUM formula in the last row of column C
5. Create a pivot table summarizing amounts by category (column A)
6. Export the result as a PDF to /workspace/report_final.pdf
7. Save and close
""")

The agent opens LibreOffice Calc, navigates the menus, selects columns, applies sorting, types formulas, creates the pivot table through the UI, and exports to PDF. Every step is guided by what it sees on screen. If LibreOffice shows a dialog ("Do you want to keep the current format?"), the agent reads it and clicks the appropriate button.

2. ERP data entry

Your ERP system (SAP, Oracle, NetSuite) has a thick-client interface or a complex web portal. Creating purchase orders, entering journal entries, or processing invoices requires navigating through multiple screens, filling forms, and clicking confirmation dialogs.

# Task: Create a purchase order in the ERP
run_desktop_task("""
1. Open Firefox and navigate to https://erp.internal.company.com
2. Log in with username 'ap_agent' and password from $ERP_PASSWORD
3. Click 'Procurement' in the left sidebar
4. Click 'Create Purchase Order'
5. Fill in:
   - Vendor: "Staples Business Advantage" (select from dropdown)
   - Delivery date: last day of next month
   - Line item 1: "Office Paper A4", Qty: 50, Unit price: 4.99
   - Line item 2: "Ink Cartridge HP 61", Qty: 10, Unit price: 24.99
6. Click 'Calculate Total'
7. Verify the total matches $499.40
8. Click 'Submit for Approval'
9. Note the PO number shown in the confirmation dialog
""")

3. Government portal form submission

Government portals are notoriously unfriendly to automation. They use custom form controls, session timeouts, CAPTCHAs, PDF uploads, and multi-step wizards with back-button-breaking JavaScript. The agent handles all of this because it interacts with the portal exactly as a human would.

# Task: File a quarterly compliance report
run_desktop_task("""
1. Open Firefox and go to https://portal.state.gov/compliance
2. Log in with the credentials in /workspace/credentials.txt
3. Click 'Quarterly Reports' > 'File New Report'
4. Select reporting period: Q1 2026
5. Upload the file /workspace/q1_compliance_data.pdf
6. Fill in the summary fields:
   - Total revenue: $2,450,000
   - Employees: 47
   - Incidents: 0
7. Check the certification checkbox
8. Click 'Submit Report'
9. Download the confirmation receipt PDF to /workspace/receipt.pdf
""")

4. Legacy Java or .NET applications

Some critical business software only runs as a desktop application. Insurance claims processors, medical records systems, and warehouse management tools often have no web version and no API. They're Java Swing apps, .NET WinForms, or even older technology.

Install the application on the desktop container and let the agent operate it. For Java apps, install the JRE. For .NET apps, use a Windows desktop type or Wine on Linux.

# Task: Process insurance claims in a Java application
run_desktop_task("""
1. The claims processing application is already running on the desktop
2. In the 'Pending Claims' list, select the first claim
3. Read the claim details: claimant name, amount, date, type
4. Cross-reference the claim against the policy data in the right panel
5. If the claim amount is under $5,000 and the policy is active:
   - Click 'Approve'
   - Enter approval code: AUTO-{date}
6. If the claim needs review:
   - Click 'Flag for Review'
   - Enter note: 'Amount exceeds auto-approval threshold'
7. Click 'Next Claim' and repeat for all pending claims
8. When no more pending claims, take a screenshot for the audit log
""")

5. Document conversion and processing

Convert documents between formats, merge PDFs, create presentations from data, or fill in PDF forms. LibreOffice handles all common office formats, and the agent operates it through the GUI.

# Task: Create a presentation from data
run_desktop_task("""
1. Open LibreOffice Impress (Presentation)
2. Create a new presentation with the 'Business' template
3. Slide 1: Title 'Q1 2026 Results', subtitle 'Sales Division'
4. Slide 2: Insert a table with these quarterly numbers:
   - North: $1.2M (+12%)
   - South: $890K (+5%)
   - East: $2.1M (+18%)
   - West: $1.5M (-3%)
5. Slide 3: Insert a bar chart visualizing the data from Slide 2
6. Slide 4: Title 'Key Takeaways', bullet points:
   - East region leading growth at 18%
   - West region needs attention, -3% decline
   - Total Q1 revenue: $5.69M (+10% YoY)
7. Save as /workspace/q1_presentation.pptx
8. Also export as PDF to /workspace/q1_presentation.pdf
""")

Orchestrating desktop tasks from a terminal sandbox

In production, the desktop container is just one part of a larger workflow. A terminal sandbox orchestrates the entire pipeline: fetch data from an API, create a desktop container to process it in a GUI application, collect the results, and push them downstream.

import asyncio, httpx, json, os

SANDBOX_URL = os.environ["SANDBOX_URL"]
API_KEY = os.environ["SANDBOX_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}


async def process_spreadsheet(input_file: str, task: str) -> str:
    """
    Create a desktop container, upload a file, run a GUI task,
    and download the result.
    """
    async with httpx.AsyncClient(timeout=300) as client:
        # 1. Create desktop
        desktop = (await client.post(
            f"{SANDBOX_URL}/api/desktops", headers=HEADERS,
            json={"desktop_type": "ubuntu-xfce", "idle_timeout_minutes": 15},
        )).json()
        cid = desktop["container_id"]

        # 2. Wait for ready
        for _ in range(40):
            status = (await client.get(
                f"{SANDBOX_URL}/api/containers/{cid}/status",
                headers=HEADERS)).json()
            if status["state"] == "assigned": break
            await asyncio.sleep(3)

        # 3. Upload the input file to the desktop
        #    (via shared volume or HTTP file transfer)

        # 4. Run the desktop agent script
        #    (SSH into the desktop or use the VNC-based agent)

        # 5. Download results

        # 6. Release desktop back to pool
        await client.post(
            f"{SANDBOX_URL}/api/containers/{cid}/release",
            headers=HEADERS)

    return "result.pdf"

The complete workstation: browser + desktop + terminal

The most powerful pattern combines all three container types in a single workflow. This is what makes the platform different from every other AI infrastructure provider: the agent gets the same workstation a human employee would get.

• Browser container — the agent's web browser. Research, web portals, SaaS applications, web-based forms.
• Desktop container — the agent's workstation. Spreadsheets, desktop applications, document processing, any GUI.
• Terminal sandbox — the agent's command line. Code execution, data processing, API calls, orchestration.

Example: end-of-month reconciliation

Every month, the finance team reconciles vendor invoices against internal records. The data lives in three places: the vendor portal (web), the accounting spreadsheet (desktop), and the internal API (terminal). Here's how the three containers work together:

1. Terminal sandbox orchestrates the workflow, queries the internal API for purchase order data, saves it as JSON.
2. Browser containers log into 10 vendor portals in parallel, download this month's invoices, save them to shared storage.
3. Desktop container opens the accounting spreadsheet in LibreOffice, enters the new invoice data, runs the reconciliation macro, generates the variance report, exports to PDF.
4. Terminal sandbox collects the PDF, emails it to the CFO, posts a summary to Slack, archives everything to S3.

No human touched any of these systems. The agent navigated web portals, operated a desktop application, and ran code — using the same interfaces a human would use.

Watching the agent work

Desktop containers have a unique advantage: you can watch the agent in real time. The access_url opens a noVNC session in your browser. You see exactly what the agent sees. The mouse moves, windows open, text gets typed. It's like watching a screen recording of an employee working — except it's happening live.

This is critical for three reasons:

• Debugging. When the agent gets stuck, you can see exactly where. It clicked the wrong menu? It can't find a button? The screenshot tells you immediately.
• Trust. The first time you deploy an AI agent on your ERP, you want to watch it. Live VNC access lets you verify that the agent is doing the right thing before you turn it loose unsupervised.
• Training. Watching the agent work reveals inefficiencies in your task description. If the agent takes 15 clicks to do something that could be done in 3, you refine the prompt.

Desktop agents vs traditional RPA

If your company already has a UiPath or Automation Anywhere deployment, you might wonder whether AI desktop agents replace it. The short answer: they solve the same problem differently, and the AI approach is better for most new automation.

When to use traditional RPA: High-volume, time-critical tasks on stable UIs where you need sub-second execution and 100% determinism. Example: processing 10,000 identical transactions per hour on an internal system that never changes.

When to use AI desktop agents: Everything else. Varied portals, changing UIs, exception handling, tasks that require judgment, tasks you need running this week not this quarter. For most enterprise desktop automation, the AI approach wins on setup time, maintenance cost, and flexibility.

What it costs

For a task that takes a human 15 minutes at $40/hour fully loaded, that's $10 of human time replaced by $0.10 of agent time. A 100× cost reduction. Run 50 desktop tasks per week and you save $25,000/year.

Production tips

Give precise task descriptions

The agent performs better when you tell it exactly what to click, not just the outcome you want. "Click the 'File' menu, then 'Export as PDF'" is better than "save it as a PDF." Precise descriptions mean fewer reasoning steps, which means faster execution and lower model costs.

Use alpine-xfce for speed, ubuntu-xfce for compatibility

Alpine-based desktops start faster and use less memory. Use them for LibreOffice tasks and simple GUI work. Switch to Ubuntu if you need to install software that requires apt-get (Java runtimes, custom .deb packages, proprietary tools).

Pre-install software in custom images

If every desktop task needs the same software (a specific Java app, a particular browser plugin), build a custom Docker image with it pre-installed. This eliminates the installation step from every task and ensures consistency.

Set resolution explicitly

Vision models work best at consistent resolutions. Set the VNC resolution to 1280×720 or 1920×1080 and don't change it between tasks. This makes the coordinates consistent and the screenshots clearer for the model.

Add delays after clicks

GUI applications take time to respond. After clicking a menu item, wait 1–2 seconds for the dropdown to appear. After clicking "Save," wait for the file dialog. The control loop should include a sleep between the action and the next screenshot to let the UI settle.

Screenshot on every failure

When the agent reports FAIL, the last screenshot shows exactly what went wrong. Save it to the workspace for debugging. This is the equivalent of a stack trace for visual automation.

Use release, not delete

Desktop containers support warm pools. After a task completes, call POST /api/containers/{id}/release to return the container to the pool. The next desktop task claims it instantly without a cold start.

Troubleshooting

Agent can't find a button that's clearly visible

The screenshot resolution might be too low for the vision model to read small text. Increase the VNC resolution or zoom in the application. Also try Claude Opus 4.6 instead of Sonnet for tasks that require reading small or low-contrast text.

Agent clicks the wrong location

Coordinate accuracy depends on the screenshot matching the actual screen state. If there's a lag between the screenshot and the click (because the UI was still animating), the agent clicks the wrong spot. Increase the wait time after screenshots. Also verify that the VNC resolution matches the screenshot resolution.

LibreOffice dialog box blocks the workflow

LibreOffice shows "Recovery" dialogs on startup if a previous session crashed. Add a pre-task step: "If you see a recovery dialog, click 'Discard' to dismiss it." Or configure LibreOffice to skip recovery prompts by editing its config files in the custom image.

Desktop container is slow to start

Desktop containers are heavier than browser containers (~2–3 GB image). First start takes 45–90 seconds in the cloud. Use warm pools to eliminate this: pre-provision 2–3 desktop containers so they're ready when you need them.

Agent runs out of steps before completing the task

The default max_steps=50 should handle most tasks. If a task genuinely requires more steps, increase the limit. But first check: does the task description have unnecessary ambiguity that's causing the agent to wander? A clearer description almost always reduces step count.

What's next