Persistent Workspaces: AI Agents That Survive the Stop

Why persistent state matters for modern AI agents

Three workloads where persistence is the difference between a working agent and a useless one:

1. Multi-day research. An agent investigating a long-running question can't restart from scratch every morning. The notes from yesterday, the sources already read, the partial conclusions — all of these need to survive the overnight stop.
2. Long-running coding. Claude Code or Devin doing a big refactor. The branch state, the failed test cache, the planning notes, the conversation history — all need to persist so day 3 isn't a re-do of day 1.
3. Always-on monitoring. An on-call agent watching a system needs continuity — what alarms fired yesterday, what's the current "normal" baseline, who's been notified. Without persistence, every restart is amnesia.

The cost driver makes persistence non-optional: always-on agents cost real money. A ab0t.medium running 24/7 is $28.80/month. Multiply by 100 agents and you're at $2,880/month for sandboxes that mostly idle. Auto-stop with persistent state brings that down 67% — and the wake latency (8-15 sec) is invisible for non-real-time workloads.

So: persistence isn't a luxury, it's the architectural requirement that makes long-running and event-driven agents economically viable. This guide is the playbook.

What survives a sandbox stop — concrete inventory

The cardinal question: when a sandbox stops, what's still there when it starts again?

Where things live, by surviving-or-not

For day-1 setup, know which directories survive:

• /data — your canonical "state-bearing" directory. Survives. Use this for agent state files.
• /workspace — primary work area. Survives. Use for active project files.
• /home/ec2-user — survives. User config (.bashrc, .gitconfig, ssh keys) lives here.
• /tmp — varies; some configurations RAM-back this. Don't trust for anything important.
• /var/log — survives. Useful for audit but can grow; rotate or delete old logs.
• /etc — survives. Any system config you write here persists.
• /proc, /sys — kernel runtime state; doesn't survive (it's regenerated on boot).
• /mnt/* — depends on the mount. EBS mounts survive; S3 mounts must be re-mounted on boot.

The checkpoint pattern

Before stopping (manually or via auto-stop), the agent serializes its state. On next start, it restores. Two implementations: simple JSON for low-stakes; sqlite for higher-stakes.

Simple JSON checkpoint

import json, datetime, os
from pathlib import Path

STATE_FILE = Path("/data/agent-state.json")

def checkpoint(state: dict):
    """Atomic write — write to .new then rename. Survives crashes mid-write."""
    STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
    tmp = STATE_FILE.with_suffix(".json.new")
    tmp.write_text(json.dumps({
        "version": 1,
        "checkpointed_at": datetime.datetime.utcnow().isoformat(),
        "state": state,
    }, indent=2))
    tmp.rename(STATE_FILE)   # atomic on POSIX

def restore():
    """Load on next start; return None if no checkpoint exists yet."""
    if not STATE_FILE.exists():
        return None
    try:
        data = json.loads(STATE_FILE.read_text())
        if data.get("version") != 1:
            raise ValueError(f"Unknown state version: {data.get('version')}")
        return data["state"]
    except (json.JSONDecodeError, KeyError) as e:
        # Corrupted state — try the backup
        backup = STATE_FILE.with_suffix(".json.bak")
        if backup.exists():
            return json.loads(backup.read_text())["state"]
        raise

Why atomic write matters: if the agent crashes mid-write, a non-atomic write leaves a half-written file. The .new + rename pattern means the file is either fully old or fully new — never half-and-half.

sqlite checkpoint (richer state)

For agents with structured state — tasks, observations, decisions, files — sqlite is more robust:

import sqlite3, json, datetime
from pathlib import Path

DB = Path("/data/agent.sqlite")

def _conn():
    DB.parent.mkdir(parents=True, exist_ok=True)
    c = sqlite3.connect(str(DB))
    c.execute("PRAGMA journal_mode=WAL")   # write-ahead log; safer + faster
    c.execute("PRAGMA synchronous=NORMAL")
    return c

def init_schema():
    with _conn() as c:
        c.execute("""
            CREATE TABLE IF NOT EXISTS tasks (
                id TEXT PRIMARY KEY,
                state TEXT NOT NULL,
                payload_json TEXT,
                created_at TEXT,
                updated_at TEXT
            )""")
        c.execute("""
            CREATE TABLE IF NOT EXISTS observations (
                id INTEGER PRIMARY KEY AUTOINCREMENT,
                task_id TEXT,
                kind TEXT,
                content TEXT,
                created_at TEXT
            )""")
        c.execute("""
            CREATE TABLE IF NOT EXISTS state_kv (
                key TEXT PRIMARY KEY,
                value_json TEXT
            )""")

def save_task(task_id, state, payload):
    now = datetime.datetime.utcnow().isoformat()
    with _conn() as c:
        c.execute("""
            INSERT OR REPLACE INTO tasks (id, state, payload_json, created_at, updated_at)
            VALUES (?, ?, ?, COALESCE((SELECT created_at FROM tasks WHERE id=?), ?), ?)
        """, (task_id, state, json.dumps(payload), task_id, now, now))

def log_observation(task_id, kind, content):
    now = datetime.datetime.utcnow().isoformat()
    with _conn() as c:
        c.execute("INSERT INTO observations (task_id, kind, content, created_at) VALUES (?, ?, ?, ?)",
                   (task_id, kind, content, now))

def get_open_tasks():
    with _conn() as c:
        return c.execute("SELECT id, state, payload_json FROM tasks WHERE state IN ('pending','in_progress')").fetchall()

Why sqlite over JSON for non-trivial state:

• Atomicity at row level (sqlite handles it; you don't have to)
• Concurrent reads from multiple processes (the agent's main loop and a status query don't conflict)
• Queryable — you can ask "show me all tasks in state X created after Y"
• Migrations supported (ALTER TABLE on schema changes)
• WAL mode means crash recovery is reliable

For most production agents with more than ~20 KB of structured state, sqlite is the right call. For simple state files (the agent's last run timestamp, a few flags), JSON is fine.

Resuming tmux sessions across stop/start

Long-running agents (research, coding) usually live inside a tmux session — partly to survive SSH disconnects, partly so the operator can attach and watch. tmux sessions don't survive a stop, but you can rehydrate them via systemd:

[Unit]
Description=AI Agent Long-Running Worker
After=network.target

[Service]
Type=forking
User=ec2-user
WorkingDirectory=/workspace
ExecStartPre=/bin/bash -c 'tmux kill-session -t agent 2>/dev/null || true'
ExecStart=/usr/local/bin/tmux new-session -d -s agent 'cd /workspace && claude --resume 2>&1 | tee -a /var/log/agent.log'
ExecStop=/usr/local/bin/tmux send-keys -t agent '/checkpoint' C-m
TimeoutStopSec=30
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

What this does:

1. On boot: ExecStartPre kills any leftover session (paranoid cleanup); ExecStart creates a fresh tmux session named agent running the agent harness.
2. On stop: ExecStop sends /checkpoint command to the agent (which the harness should interpret as "save state and exit cleanly"). TimeoutStopSec=30 gives 30 seconds for the agent to finish before forced kill.
3. On crash: Restart=always with RestartSec=10 automatically restarts. With persistent state, this is non-disruptive.

Enable once with systemctl enable agent.service. After every sandbox boot, the agent comes back up automatically.

The agent's resume hook

Most modern agent harnesses support a resume hook. Concrete patterns:

• Claude Code — ~/.claude/memory.md is the long-term memory. Lives on the persistent workspace; reads on every startup. Plus --resume CLI flag for picking up the last conversation.
• Codex CLI — .codex/state.json in the project directory. Restored on startup.
• Devin — fully managed; resume is automatic.
• Cursor Cloud Agents — workspace state preserved across sessions; cursor manages the resume.
• Custom agents — your responsibility. Pattern: agent's main() first calls restore(); uses returned state if present, else starts fresh.

The cost math: always-on vs cycle-stopped

Auto-stop is essentially free leverage. The disk stays around (and so does all the agent's state, packages, and history); only the compute meter pauses. The trade-off: 8-15 second resume latency when work arrives. For event-driven agents, that's invisible to the operator.

Storage cost analysis

Storage cost scales linearly with workspace size. At $0.10/GB-month:

For workloads above 100 GB, consider whether mounting S3 (no per-GB storage charge for sandbox; S3 storage at $0.023/GB-month for Standard) is cheaper. Crossover for read-mostly: ~200 GB. Read more in File I/O.

The multi-day research / coding agent pattern

Concrete: a Claude Code research agent investigating a long-running question. Days 1-3 it browses, takes notes, writes scratch files. Day 4 it compiles a report. Auto-stop kicks in between work bursts; on resume, the agent reads its scratch notes and continues.

Workspace layout

/workspace/
├── decisions.md           # "what I tried, what worked, what didn't" — narrative log
├── plan.md                # current plan; agent updates as it learns
├── notes/                 # findings organized by topic
│   ├── topic-a.md
│   ├── topic-b.md
│   └── ...
├── sources/               # downloaded source material (PDFs, web pages)
│   ├── source-001.pdf
│   ├── source-002.html
│   └── ...
├── output/                # draft + final reports
│   ├── draft-v1.md
│   ├── draft-v2.md
│   └── final-report.pdf
└── .git/                  # yes, git the workspace — version control everything

/data/
├── agent-state.json       # current task, context window summary, todo list
├── agent.sqlite           # structured state if richer than JSON
└── session.tmux           # tmux session config

/var/log/
└── agent.log              # systemd captures all stdout/stderr

The resume protocol

Every time the agent starts (cold or resume from auto-stop), its first action is "read context to orient." Concretely:

def orient_on_start():
    """Run as the first action on every agent boot."""
    state = restore() or {"first_run": True}

    # 1. Read the persistent docs first
    decisions = Path("/workspace/decisions.md").read_text() if Path("/workspace/decisions.md").exists() else ""
    plan = Path("/workspace/plan.md").read_text() if Path("/workspace/plan.md").exists() else ""

    # 2. Get current open tasks from sqlite
    open_tasks = get_open_tasks()

    # 3. Build the orientation prompt
    orientation = f"""
You are resuming work on a multi-day research project.

PRIOR DECISIONS (from /workspace/decisions.md):
{decisions[-2000:]}

CURRENT PLAN (from /workspace/plan.md):
{plan}

OPEN TASKS:
{format_tasks(open_tasks)}

LAST CHECKPOINT: {state.get('checkpointed_at', 'never')}

Read these, then continue from where you left off.
"""
    return orientation

The agent calls orient_on_start(), sees its prior context, and continues without re-doing day 1's work.

Daily snapshot

Every day at end-of-business, the agent (or a cron job) snapshots /workspace to S3:

#!/bin/bash
# /etc/cron.daily/agent-snapshot
set -euo pipefail

DATE=$(date +%Y-%m-%d)
ARCHIVE="/tmp/snapshot-${DATE}.tar.gz"
S3_BUCKET="agent-snapshots"
AGENT_NAME="research-agent-001"

# Tar workspace + state, exclude noisy bits
tar --exclude='/workspace/sources/*.tmp' \
    --exclude='/workspace/.git/objects/pack' \
    -czf "$ARCHIVE" /workspace /data

# Upload
aws s3 cp "$ARCHIVE" "s3://${S3_BUCKET}/${AGENT_NAME}/${DATE}/snapshot.tar.gz" \
  --storage-class STANDARD_IA \
  --sse AES256

# Cleanup
rm "$ARCHIVE"

# Notify
curl -X POST $SLACK_WEBHOOK_URL \
  -d "{\"text\": \"Snapshot ${DATE} for ${AGENT_NAME} complete\"}"

Lifecycle on the S3 bucket: keep snapshots for 90 days, transition to GLACIER at 30 days, delete at 90. Storage cost is negligible; recovery option is reliable.

Monthly clean restart

Even with persistent state, restart the agent monthly to clear any accumulated cruft. Like restarting a long-running server. The platform supports configurable scheduled restarts.

Pattern that works:

1. 1st of each month at 3am UTC: cron triggers systemctl stop agent
2. Sandbox is paused (auto-stop policy)
3. Within an hour: scheduled wake fires; sandbox starts; agent.service restarts
4. Agent reads its state, continues from checkpoint

The agent gets a "fresh process" without losing state. Catches accumulated memory leaks, file-descriptor leaks, in-RAM cruft.

When persistent workspaces are wrong

Not every agent needs persistence. Anti-patterns:

• Stateless one-shot tasks (process this invoice, summarize this PDF). Use ephemeral containers — they spin up faster and have nothing to maintain.
• Tasks where stale state is dangerous (the agent's "memory" might be of a system that has since changed). Prefer fresh-every-time + read-the-source-of-truth.
• High-security workloads where you want zero residue between runs. Ephemeral.
• Multi-tenant agents serving many customers — you don't want one customer's state leaking to another. Per-task ephemeral.
• Highly bursty workloads where the average idle time is days. Storage cost accumulates; consider just regenerating state from canonical sources on each run.
• Workloads where the sandbox is expensive (g4dn.* with GPU) and idle is unpredictable. Auto-stop saves compute but storage stays; if you idle 90% of the time, the platform's "stop and detach disk" mode (move state to S3, re-mount on resume) might be cheaper.

Backup, snapshot, and disaster recovery

The DR strategy in three layers

1. Daily S3 sync. Critical state files copy to S3 once a day. Survives sandbox loss.
2. Versioning on the bucket. Catch the "agent corrupted its state" case — roll back to yesterday.
3. Cross-region replication. For mission-critical: replicate the snapshot bucket to a secondary region.

Quarterly DR drill

Untested backups don't exist. Once a quarter:

1. Pick an agent (rotate through the fleet).
2. Terminate its sandbox.
3. Provision a fresh sandbox of the same size.
4. Pull the most recent snapshot from S3; extract to /workspace + /data.
5. Start the agent service.
6. Verify the agent picks up correctly: its first message references prior state; its first action is sensible given the history.
7. Document the drill (timestamp, agent, recovery time, any issues found).

Track recovery time. If it's growing over time, the backup process is degrading; investigate. Operators we work with target ≤30 minutes for full recovery on a ab0t.medium agent with 10 GB workspace.

Audit log lives separately

The audit log is in the platform's managed store, not your sandbox. This is intentional — the audit trail must survive sandbox compromise or accidental deletion. For compliance, configure both:

• Sandbox-disk persistence for agent state and work product
• Platform audit log (separate retention) for the action history

Together they form a recoverable, audit-grade record of everything the agent did.

Recovering from state corruption

What if the state file gets corrupted mid-write? Three layers of defense:

Atomic writes

Always write to X.new then rename to X. POSIX rename is atomic; the file is either fully old or fully new. Never write directly to X.

Versioned state files

Keep last N versions:

import shutil, datetime
from pathlib import Path

STATE = Path("/data/agent-state.json")
HISTORY = Path("/data/state-history")
KEEP_VERSIONS = 10

def checkpoint_versioned(state: dict):
    HISTORY.mkdir(parents=True, exist_ok=True)
    # Archive current state before overwriting
    if STATE.exists():
        ts = datetime.datetime.utcnow().strftime("%Y%m%dT%H%M%SZ")
        shutil.copy(STATE, HISTORY / f"state-{ts}.json")
    # Write new state atomically
    tmp = STATE.with_suffix(".json.new")
    tmp.write_text(json.dumps(state))
    tmp.rename(STATE)
    # Trim history
    versions = sorted(HISTORY.glob("state-*.json"), reverse=True)
    for old in versions[KEEP_VERSIONS:]:
        old.unlink()

If today's state is bad, you have yesterday's. If yesterday's is bad, the day before. Cheap insurance.

S3 backup as last resort

Daily S3 sync (above) means even total filesystem loss is recoverable. Cross-region replication on the S3 bucket means even region-level loss is recoverable.

Migrating between sandbox sizes

Workloads grow. The agent that was fine on a ab0t.micro may need ab0t.medium six months in. The platform supports in-place resize:

1. Stop the sandbox: POST /api/sandboxes/{id}/stop
2. Resize: PATCH /api/sandboxes/{id} with {"instance_type": "ab0t.medium"}
3. Start: POST /api/sandboxes/{id}/start

Disk and state survive. The agent restarts with more RAM / CPU. Down-sizing works the same way (with the caveat that you must verify the agent fits in the smaller memory).

Things to verify after resize:

• Agent has the memory it needs (check free -h)
• No hardcoded references to instance type (in CLAUDE.md, scripts, config)
• Performance hasn't unexpectedly changed (compare time-to-completion before/after)

Multi-agent shared workspace

Sometimes you want multiple agents to share state — a fleet of researchers all updating the same notes folder; a team of code reviewers sharing a model of the codebase. Three patterns:

Shared EBS volume

Mount the same EBS volume read-write across multiple sandboxes (with file-locking discipline). Cheap, simple. Caveat: EBS supports multi-attach but not write-anywhere — you need application-level locking.

Shared NFS / EFS

For more concurrency, use AWS EFS (NFS for AWS). Multiple agents mount; concurrent writes work via standard NFS semantics. ~3× the cost of EBS but supports many concurrent writers. Use for fleet-wide state where conflicts are common.

Shared sqlite (single-writer model)

One agent owns the sqlite; others connect via network (e.g. via litestream or rqlite). Cleaner concurrency story than direct file-share. Best for "fleet has structured shared state" — tasks queue, observation log, decision history.

Comparison vs E2B / Modal / Daytona / Browserbase

For AI agent workloads specifically, the patterns in this guide work across most of these platforms. Sandbox Platform's differentiation: per-agent isolation by default, integrated audit log, integrated cost-cap budgeting, and the auto-stop/storage cost ratio is among the cheapest in the market.

Real-world patterns from the field

Pattern 1: 30-day research agent

A consulting firm. Single Claude Code agent investigating a market. Workspace at 12 GB after 30 days (downloaded sources, notes, drafts). Auto-stop active; agent runs 4-6 hours/day. Cost: ~$15/month compute + $1.20 storage = $16.20. Daily snapshot to S3 ($0.005/day storage). Quarterly DR drill verifies recovery in 12 minutes.

Pattern 2: Always-on on-call agent

A SaaS company. Single ab0t.small persistent sandbox; agent watches alert streams 24/7. Workspace at 2 GB (alert history, runbook docs). No auto-stop (always-on). Cost: $14/month compute + $0.20 storage. Restarts monthly via cron at 4am Sunday. State snapshots every 6 hours to S3.

Pattern 3: ML training pipeline

An ML team. Fleet of ab0t.gpu agents preprocessing data + training. Workspaces at 200 GB each (model weights, intermediate datasets). Auto-stop active during preprocessing phases (which idle between batches); always-on during training. Cost varies $130-380/month per agent depending on training cadence. Models replicated to S3 every 2 hours during training.

Pattern 4: Multi-day code refactor

An engineering team. Devin running a large refactor over 5 days. Workspace at 8 GB (cloned repo, partial work, test results). Auto-stop active; agent runs 6-10 hours/day. Cost: ~$45/month compute. Branch state survives auto-stop; final PR opened on day 5.

Pattern 5: Long-running competitive intel

A marketing agency. Single ab0t.micro running a continuously-updating model of competitor moves. Workspace at 5 GB (history of every observed change). Auto-stop active during low-activity hours (12am-5am). Cost: $4/month compute + $0.50 storage. Restarts quarterly with full state preserved.

Anti-patterns: what not to do

• Storing state only in RAM. First sandbox stop kills it. Always serialize.
• Writing state non-atomically. Crash mid-write leaves a half-file. Use the .new+rename pattern.
• No backup off the sandbox. Sandbox loss = state loss. Daily S3 sync is mandatory.
• Running everything as root. File permissions exist for a reason. Run as ec2-user; reserve root for installs.
• Storing secrets in /data. Use a secrets manager (AWS Secrets Manager, Doppler, 1Password). Never commit secrets to a workspace.
• Trusting /tmp for anything important. Some configs RAM-back /tmp; survives stop, doesn't survive reboot. Use /data instead.
• Letting state files grow unbounded. Rotate, archive, prune. A 50 GB state file is harder to checkpoint and harder to back up than a 50 MB state file with off-loaded history.
• No DR drill. Backups you haven't tested don't work.
• Confusing audit log with backup. Audit log is "what happened"; backup is "the state to restore." Both are needed; they're different.
• Checkpointing too rarely. Auto-stop fires; if your last checkpoint was 4 hours ago, you lose 4 hours of work. Checkpoint at every meaningful transition.

Frequently asked questions

How long can a sandbox stay stopped before being deleted?

Indefinitely, by default. Stopped sandboxes don't accrue compute charges; storage continues at $0.10/GB-month. If you want stopped sandboxes auto-deleted after N days of idle, configure that at the workspace level.

Does the public IP change on resume?

By default, no — the platform holds the same Elastic IP. If you configure "release IP on stop", you get a new public IP on each resume. Most operators leave it stable.

What about the model's context window — does that persist?

No. The model's context window is conversation history fed into each call; lives in RAM during a run and disappears after. To "remember" across runs, the agent has to write to disk explicitly. Claude Code uses ~/.claude/memory.md; Codex CLI uses .codex/state.json.

Can multiple agents share a workspace?

Yes — for "agent fleets working on the same project." Mount the same EBS volume read-write (with file-locking), or use a shared NFS mount. Heavy concurrent writes? Use distinct subdirectories per agent and aggregate periodically.

Do persistent workspaces work with warm pools?

Different patterns for different needs. Warm pools are for low-latency, identity-less workloads. Persistent workspaces are for identity-bearing workloads. They don't mix — a workspace can't be "warm" because the warmth is the state, and the state is per-agent.

How do I do disaster recovery for an agent's persistent state?

Daily S3 sync of /data and /workspace. Versioned bucket. Quarterly DR drill: terminate, restore, verify the agent picks up correctly.

What if the agent's state file gets corrupted?

Three layers: atomic write, versioned files, S3 backup. For high-stakes workloads, use sqlite with WAL mode — it handles atomicity natively.

Can I migrate an agent from a ab0t.medium to a ab0t.large without losing state?

Yes. The platform supports in-place resize: stop, change instance type, start. Disk and state survive.

What happens if I delete a sandbox by mistake?

Soft-deletes are recoverable for 30 days by default (workspace-configurable). Hard-deletes are not — that's why DR backups matter.

How big can a persistent workspace get?

Default disk is 30 GB; configurable up to 16 TB. For workloads above 100 GB, consider mounting S3 instead of storing locally.

Does sandbox auto-stop cost me anything in latency?

8-15 second wake from auto-stop. Invisible to the human on the other end of an event-driven flow. For latency-sensitive interactive flows, use warm pools.

How do I version-control my agent's CLAUDE.md across stop/start?

Two patterns. (1) Keep CLAUDE.md in a git repo on the sandbox; commit changes. (2) Keep CLAUDE.md in workspace settings (off-sandbox); platform tracks history. The platform option is better for compliance.

What about cross-region replication of agent state?

S3 cross-region replication on your snapshot bucket. Plus: deploy the agent in the secondary region with same template; resume from snapshot if primary fails. Most operators don't need this.

Can I encrypt the persistent workspace?

Yes. EBS volumes use AWS-managed KMS by default; configurable to customer-managed KMS. Files written to S3 inherit bucket SSE settings.

What's the difference between persistent workspace and stateful agent?

Persistent workspace is the underlying disk that survives stop/start. Stateful agent is the higher-level pattern of an agent that maintains memory / context across runs. Persistent workspace enables stateful agents but doesn't make them stateful by itself — the agent still has to serialize state explicitly.

What's next