Event Architecture Deep Dive

Why event-driven, why now

The shift from request-driven to event-driven AI agents in 2026 is the architectural correlate of the shift from "AI as a tool" to "AI as employee." A human employee doesn't sit at their desk waiting for a manager to type a prompt. They react to events: an email arrives, a calendar reminder fires, a Slack message pings, a deploy completes, a customer files a ticket, a spreadsheet gets updated, a deadline approaches. AI employees work the same way.

Three forces drove this shift in 2025-2026:

1. Compute economics. Always-on agents cost real money — a ab0t.medium running 24/7 is $28.80/month. Multiply by 100 agents and you're at $2,880/month for sandboxes that mostly idle. Auto-stop with event-driven wake brings that down 67% — and the wake latency (8-15 sec) is invisible to humans on the other end.
2. Mental model. Operators stopped thinking "I prompt the agent" and started thinking "the agent has an inbox." Slack-driven, email-driven, calendar-driven workflows are how employees feel; the API-call-driven workflow is how libraries feel. Customers want the former.
3. Reliability. A polling agent is a single point of failure. An event-driven agent with a queue and DLQ has natural retry, replay, and observability. Production teams who tried polling-based architectures in 2024 mostly migrated by 2026.

This guide is the wiring playbook. Code, diagrams, cost math. Use it when you're building production event-driven agents — not when you're prototyping (for prototypes, polling is fine).

The reference architecture

Five components:

1. Event source — Slack, GitHub, Stripe, Linear, your own systems, AWS services emitting to EventBridge, cron schedules, calendar entries.
2. Receiver — a thin Lambda / FastAPI / Cloudflare Worker that validates the event (HMAC, signature, schema), dedups (idempotency key), and pushes to a queue. Should respond 200 in under 1 second; long work happens downstream.
3. Queue — SQS, EventBridge, Redis Streams, RabbitMQ, NATS. Buffers events; supports retry with backoff; supports DLQ for poison messages.
4. Agent sandbox — your AI worker, running on Sandbox Platform. Wakes on event arrival, processes, posts results, returns to idle. Serializes state for resume.
5. Output — wherever the operator sees the result: Slack thread, Linear ticket, email digest, database write, downstream API call.

Plus: a dead-letter queue (DLQ) for events that fail processing N times, a cron / scheduler for time-driven triggers, and an audit log capturing every step (which most platforms ship for free; check yours).

Trigger type 1: Webhooks

Webhooks are how SaaS systems push events to you. Slack mention, GitHub PR opened, Stripe charge failed, Linear issue created — each fires a webhook to a URL you provide. The pattern across all of them is similar; the gotchas are subtle.

A production-grade webhook receiver

import hashlib, hmac, time, os, json
from fastapi import FastAPI, Request, Header, HTTPException
import boto3

app = FastAPI()
sqs = boto3.client("sqs")
QUEUE_URL = os.environ["AGENT_QUEUE_URL"]
SLACK_SIGNING_SECRET = os.environ["SLACK_SIGNING_SECRET"]
SLACK_SIGNING_SECRET_PREVIOUS = os.environ.get("SLACK_SIGNING_SECRET_PREVIOUS")

def verify_slack_signature(body: bytes, timestamp: str, signature: str) -> bool:
    """Verify against current secret, fall back to previous (rotation window)."""
    # Reject events older than 5 minutes (replay-attack guard)
    if abs(time.time() - int(timestamp)) > 300:
        return False

    sig_basestring = f"v0:{timestamp}:".encode() + body
    for secret in [SLACK_SIGNING_SECRET, SLACK_SIGNING_SECRET_PREVIOUS]:
        if not secret:
            continue
        expected = "v0=" + hmac.new(secret.encode(), sig_basestring, hashlib.sha256).hexdigest()
        if hmac.compare_digest(expected, signature):
            return True
    return False

@app.post("/webhooks/slack")
async def slack_webhook(
    request: Request,
    x_slack_request_timestamp: str = Header(...),
    x_slack_signature: str = Header(...),
):
    body = await request.body()

    # 1. Verify signature
    if not verify_slack_signature(body, x_slack_request_timestamp, x_slack_signature):
        raise HTTPException(status_code=401, detail="invalid signature")

    payload = json.loads(body)

    # 2. Handle URL verification challenge (Slack's setup ping)
    if payload.get("type") == "url_verification":
        return {"challenge": payload["challenge"]}

    # 3. Push to queue with idempotency key
    event_id = payload.get("event_id") or payload.get("event", {}).get("client_msg_id")
    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps({"source": "slack", "event": payload}),
        MessageDeduplicationId=event_id,           # for FIFO queues
        MessageGroupId=payload.get("team_id", "default"),
        MessageAttributes={
            "event_id": {"DataType": "String", "StringValue": event_id},
        },
    )

    # 4. Respond fast — Slack expects 2xx within 3 seconds
    return {"ok": True}

Five things this receiver does that most don't:

1. HMAC verification with a 5-minute timestamp window. The timestamp check protects against replay attacks — an attacker who captured a valid signed payload can't replay it 10 minutes later.
2. Dual-secret acceptance for rotation. Both SLACK_SIGNING_SECRET and SLACK_SIGNING_SECRET_PREVIOUS are checked. During rotation, push the new secret as primary; the previous one stays as fallback for 24 hours; then retire the previous.
3. Constant-time comparison via hmac.compare_digest. A regular == would leak signature bytes through timing. compare_digest takes constant time regardless of where the comparison fails.
4. URL verification challenge. Slack's first webhook to your URL is a url_verification ping — you echo back the challenge. Many setups break here because the receiver tries to dedup the verification or push it to a queue.
5. Queue handoff with idempotency. The receiver does the minimum (verify, push) and returns 200 fast. Long work happens downstream. Slack times out webhooks at 3 seconds; if you take longer, Slack retries and you get duplicate processing.

Replay attacks and what they look like

An attacker captures a valid signed webhook (e.g. via a misconfigured CDN log, a compromised intermediate proxy, or social-engineering an employee to forward an email containing the payload). Without a timestamp check, the attacker replays it tomorrow and your receiver believes it's a fresh event. Defense:

• Timestamp window. Reject events older than 5 minutes. Slack, GitHub, Stripe, and Linear all include a timestamp in the signature payload.
• Idempotency cache. Cache delivery IDs for 24 hours. Reject duplicates even within the timestamp window. Belt + suspenders.
• HTTPS only. Webhook URLs should be HTTPS. Senders refuse to deliver to HTTP in 2026.
• Restrict the receiver's accept list. Slack publishes IP ranges; restrict the receiver's ALB security group to those ranges.

The idempotency cache

Webhook senders deliver at-least-once, not exactly-once. If your receiver returns 5xx (or doesn't respond in 3 seconds), the sender retries. Without dedup, you process the same event twice — the agent posts the same Slack reply twice, the AP clerk pays the same invoice twice.

import redis
r = redis.Redis(host="cache", decode_responses=True)

def already_processed(event_id: str) -> bool:
    """SETNX with 24h TTL — atomic dedup. Returns True if duplicate."""
    key = f"event:{event_id}"
    was_new = r.set(key, "1", ex=86400, nx=True)
    return not was_new

# In your handler:
if already_processed(event_id):
    return {"ok": True, "deduplicated": True}

Why SETNX matters: it's atomic. Two webhook deliveries arriving 10ms apart both check the cache — without atomicity, both might see "not present" and both process. SETNX guarantees exactly one wins.

Redis is the standard cache choice; sqlite-on-disk works for small deployments; DynamoDB with TTL works for AWS-native deployments without Redis. Pick by your stack.

Secret rotation without downtime

You'll rotate webhook signing secrets quarterly (best practice) or after an incident (security requirement). The dual-secret pattern from above handles this:

1. Day 1: Generate new secret. Push to receiver as SLACK_SIGNING_SECRET; current secret moves to SLACK_SIGNING_SECRET_PREVIOUS. Receiver accepts both.
2. Day 1: Update Slack's webhook configuration with the new secret. Slack starts signing with the new secret; receiver accepts via the primary slot.
3. Day 1-7: In-flight webhooks signed with the old secret continue to verify via the previous slot.
4. Day 7+: Retire the previous slot. Set SLACK_SIGNING_SECRET_PREVIOUS to empty. Only the new secret is accepted.

For multi-instance receivers, push the secret update to all instances simultaneously (config-management tool, or restart with new env). Don't rotate in a partial state where some receivers know the new secret and others don't.

Real services and their quirks

Each service ships a verification helper in their official SDK; you almost always want to use it rather than implementing your own. The boilerplate above is for cases where the SDK isn't available or you want explicit control.

Common webhook failure modes

• Receiver returns 5xx. Sender retries; you process duplicates. Fix: dedup with idempotency keys.
• Receiver returns 200 but pushes to dead queue. Sender thinks delivery succeeded; events vanish. Fix: monitor queue ingress + a dead-letter queue.
• Receiver takes longer than the sender's timeout. Sender retries. Fix: respond 200 fast; long work in queue.
• HMAC fails because of a proxy URL rewrite. Some senders sign the URL; if a proxy rewrites the URL between the sender and your receiver, signature verification fails. Fix: terminate TLS at the receiver or configure the proxy to preserve the original URL.
• Webhook delivery to your URL but no event in your queue. Usually means the receiver rejected (HMAC failed, schema validation failed, dedup hit). Check the receiver's logs.
• Stale signing secret in env after rotation. Receiver returns 401; sender retries; queue stays empty. Fix: alert on 401 rate; deploy secret updates atomically.

Trigger type 2: SQS / EventBridge

Webhooks are the inbound event surface. Queues are the buffering layer between inbound and your agents. They give you retry, replay, observability, and decoupling.

SQS vs EventBridge — when to use which

Most AI agent fleets we work with use both: EventBridge as the front-door router (events from Stripe, GitHub, Slack, internal services all land here) + SQS as the per-agent worker queue (EventBridge rule routes matching events into the agent's SQS queue, agent pulls and processes).

Visibility timeout: the most-misunderstood SQS setting

When a worker pulls a message from SQS, the message becomes invisible to other workers for the visibility timeout duration. If the worker finishes and acks the message within the timeout, the message is deleted. If the worker crashes or takes too long, the message reappears on the queue and a different worker picks it up.

Three failure modes:

• Timeout too short: worker is mid-task, message reappears, another worker grabs it, you have two workers doing the same task. The kind of duplicate processing that bites in production.
• Timeout too long: a crashed worker's task sits invisible for hours before another worker can retry. Wasted time.
• Variable task duration: some tasks take 30 sec, some take 5 minutes. Pick a timeout for the worst case, or use ChangeMessageVisibility to extend dynamically.

The right pattern:

import threading, time, boto3
sqs = boto3.client("sqs")

def extend_visibility(queue_url, receipt_handle, stop_event, interval=30):
    """Background thread: extend visibility every {interval}s while task runs."""
    while not stop_event.is_set():
        time.sleep(interval)
        if stop_event.is_set():
            break
        sqs.change_message_visibility(
            QueueUrl=queue_url,
            ReceiptHandle=receipt_handle,
            VisibilityTimeout=60,   # extend by 60 sec each tick
        )

def process_message(msg):
    stop = threading.Event()
    extender = threading.Thread(
        target=extend_visibility,
        args=(QUEUE_URL, msg["ReceiptHandle"], stop),
        daemon=True,
    )
    extender.start()
    try:
        # Do the long task — could take 30 sec or 30 min
        run_agent(msg["Body"])
        sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg["ReceiptHandle"])
    finally:
        stop.set()
        extender.join(timeout=5)

The background thread keeps the message invisible while the worker actually works on it. If the worker crashes, the thread stops, the message reappears at the natural timeout, and another worker picks it up. Standard production pattern.

Dead-letter queues

Some messages will fail repeatedly — bad input, downstream system permanently broken, an agent hallucinating in a loop. After N retries, the message goes to a dead-letter queue (DLQ) for human triage instead of looping forever.

resource "aws_sqs_queue" "agent_dlq" {
  name                       = "agent-events-dlq"
  message_retention_seconds  = 1209600   # 14 days
}

resource "aws_sqs_queue" "agent_events" {
  name                       = "agent-events"
  visibility_timeout_seconds = 450       # 7.5 min — 1.5x worst-case task
  message_retention_seconds  = 86400     # 24 hours

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.agent_dlq.arn
    maxReceiveCount     = 3                  # 3 retries then DLQ
  })
}

What to do with the DLQ:

• Daily digest: at 9am, post DLQ size + category breakdown to Slack. "DLQ has 3 messages: 2 portal-login failures, 1 invoice format issue. Triage when convenient."
• Manual replay: a "replay this message" button or CLI that pulls from DLQ and pushes back to the main queue.
• Pattern detection: if 5+ failures cluster around the same downstream system, that's a vendor issue, not an agent issue. Fix at the source.
• Retention: 14 days in the DLQ, then auto-delete. Long enough for triage; short enough to bound storage.

Exactly-once vs at-least-once

The Two Generals problem says exactly-once delivery is theoretically impossible across distributed systems. In practice, you have three real options:

For 95% of AI agent workloads, at-least-once with idempotent receivers is the right answer. The agent's task should be safe to run twice (idempotent operations like upserts, "already done" checks, etc.). When the task isn't naturally idempotent (sending an email, charging a card), wrap it in an idempotency-key check.

Trigger type 3: Cron and scheduled fires

Scheduled triggers are the simplest event source on paper: at time X, fire event Y. In practice, cron is the most common silent-failure surface in production AI agent fleets. Three failure modes:

The three cron failure modes

1. Host reboot during the cron tick. Bare crontab fires only when the host is up. If the host reboots at 6:00am and your job is scheduled for 6:00am, the job silently doesn't fire. Cloud schedulers (EventBridge Schedules, Cloud Scheduler) survive host failures.
2. DST and timezone confusion. "Every weekday at 6am Eastern" runs into DST transitions where 6am local time happens 0 or 2 times in a day. Use UTC schedules with explicit timezone-conversion at the receiver, or use a scheduler that handles DST natively.
3. Job overran the next tick. A nightly job scheduled for 2am that takes 90 minutes blocks the 3am tick. Either run jobs in their own sandboxes (always safe) or use a scheduler that supports overlap policy ("skip if previous run is still active").

Cloud schedulers vs systemd timers vs bare crontab

A cron-fire receiver pattern

import json, datetime, os, boto3
sqs = boto3.client("sqs")
QUEUE_URL = os.environ["AGENT_QUEUE_URL"]

def cron_handler(event, context):
    """Lambda invoked by EventBridge Scheduler at the configured cadence."""
    schedule_name = event.get("schedule_name", "unknown")
    fire_time = event.get("fire_time", datetime.datetime.utcnow().isoformat())

    # Idempotency: schedule_name + fire_time uniquely identifies a tick
    event_id = f"cron:{schedule_name}:{fire_time}"

    sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps({
            "source": "cron",
            "schedule": schedule_name,
            "fire_time": fire_time,
            "event_id": event_id,
        }),
        MessageDeduplicationId=event_id,
        MessageGroupId=schedule_name,
    )

    # Emit metric so we can detect missed fires
    boto3.client("cloudwatch").put_metric_data(
        Namespace="AgentFleet",
        MetricData=[{
            "MetricName": "CronTickFired",
            "Dimensions": [{"Name": "Schedule", "Value": schedule_name}],
            "Value": 1,
            "Timestamp": datetime.datetime.utcnow(),
        }]
    )

Pair this with a CloudWatch alarm: "alarm if CronTickFired metric for schedule X is missing for 1.5 × the schedule period". That catches the silent-no-fire case.

Overlap policies for long-running scheduled jobs

If your competitive-intel sweep takes 90 minutes and runs every hour, you'll have overlapping runs. Three policies:

• Skip overlap — if the previous run is still active, don't fire the next. Simplest; loses a tick if jobs go long.
• Queue overlap — fire normally; rely on the worker queue to serialize. Workers process them in order. The 1pm tick may not start until 2:30pm because 12pm overran.
• Concurrent fire — fire every tick into independent sandboxes. Allows parallel execution; only safe if each run is independent.

The right policy depends on whether your jobs are stateless (concurrent is fine) or stateful (need serialization). Most AI agent workloads we see are stateless per-tick — concurrent fire is the right default.

Trigger type 4: Calendar-aware agents

The most underrated event source in 2026: the agent's own calendar. A human employee knows that quarter-end compliance is due March 31st. They know the board meeting is Thursday so the metrics dashboard needs updating by Wednesday. They plan their work around upcoming deadlines.

AI employees with calendar access do the same.

Google Calendar integration

from google.oauth2.credentials import Credentials
from googleapiclient.discovery import build
import datetime

def get_upcoming_events(creds: Credentials, days_ahead=7):
    """Pull upcoming events from the agent's own calendar."""
    service = build("calendar", "v3", credentials=creds)
    now = datetime.datetime.utcnow()
    end = now + datetime.timedelta(days=days_ahead)

    events = service.events().list(
        calendarId="primary",                # the agent's own calendar
        timeMin=now.isoformat() + "Z",
        timeMax=end.isoformat() + "Z",
        singleEvents=True,
        orderBy="startTime",
    ).execute().get("items", [])

    return events

# In the agent's morning routine:
events = get_upcoming_events(agent_creds, days_ahead=7)
for e in events:
    if e["summary"].lower().startswith("month-end"):
        # Plan the work backward from the deadline
        plan_month_end_reconciliation(e["start"])

Auth pattern: each AI employee gets its own Google Workspace account (nina@yourcompany.com). The account has its own calendar that operators populate with deadlines, recurring obligations, and one-off tasks. The agent reads its calendar like a human would — daily, in the morning, planning around upcoming events.

iCal feeds (when Google Calendar isn't an option)

For non-Google-Workspace deployments, iCal feeds work. Most calendar providers (Apple iCloud, Outlook, Office 365, Fastmail) emit iCal feeds. Your agent subscribes to the feed and parses it on a schedule.

import requests
from icalendar import Calendar
import datetime

def parse_ical_feed(feed_url, days_ahead=7):
    cal = Calendar.from_ical(requests.get(feed_url).text)
    now = datetime.datetime.utcnow()
    end = now + datetime.timedelta(days=days_ahead)
    events = []
    for component in cal.walk("VEVENT"):
        start = component.get("DTSTART").dt
        if isinstance(start, datetime.datetime) and now <= start <= end:
            events.append({
                "summary": str(component.get("SUMMARY")),
                "start": start,
                "description": str(component.get("DESCRIPTION") or ""),
                "uid": str(component.get("UID")),
            })
    return events

iCal is one-way (read-only). For two-way (the agent creates events on the calendar), Google Calendar API or CalDAV is required.

The deadline-driven planning pattern

Calendar isn't just "fire on event start." It's a planning input. The agent reads its calendar, identifies upcoming deadlines, plans backward from them.

DEADLINE_BUFFERS = {
    "month-end-reconciliation": datetime.timedelta(days=2),
    "quarterly-board-prep": datetime.timedelta(days=5),
    "compliance-filing": datetime.timedelta(days=3),
    "weekly-client-digest": datetime.timedelta(hours=4),
}

def plan_around_deadlines(events):
    """For each upcoming deadline, schedule the work to start at deadline - buffer."""
    plan = []
    for e in events:
        category = classify_event(e["summary"])
        if category in DEADLINE_BUFFERS:
            buffer = DEADLINE_BUFFERS[category]
            start_at = e["start"] - buffer
            plan.append({
                "task": f"prepare-{category}",
                "deadline": e["start"],
                "start_at": start_at,
                "event_uid": e["uid"],
            })
    return plan

The agent runs this every morning. New tasks get added to its work queue with a future start_at time. The cron / scheduler fires those tasks when their start time arrives. Combined with the agent's own scheduling, this lets it work like a calendar-aware human assistant.

Long-running agent lifecycle

Some agents are not event-driven — they run continuously, maintaining state across days. Research agents investigating a long-running question, on-call agents watching a system, monitoring agents tracking pricing changes. Their lifecycle has different shape:

Health endpoints

The agent exposes an HTTP /healthz endpoint that returns 200 if it's processing events normally, 503 if it's stuck. The platform's monitoring polls every 30 seconds; alerts on 5+ consecutive failures.

from fastapi import FastAPI
import time

app = FastAPI()
agent_state = {"last_event_processed": time.time()}

@app.get("/healthz")
async def health():
    seconds_idle = time.time() - agent_state["last_event_processed"]
    # Idle for >10 min during business hours = stuck
    if seconds_idle > 600 and is_business_hours():
        return {"status": "degraded", "seconds_idle": seconds_idle}, 503
    return {"status": "healthy", "seconds_idle": seconds_idle}

Graceful shutdown on SIGTERM

Sandbox Platform sends SIGTERM 30 seconds before forcibly stopping a sandbox (during scaling, host migration, or operator-triggered stop). The agent handles SIGTERM by checkpointing and exiting cleanly.

import signal, json, sys, time

def handle_sigterm(signum, frame):
    # Grace period: 30 seconds to flush state
    print("SIGTERM received; checkpointing...")
    state = collect_current_state()
    with open("/data/agent-state.json", "w") as f:
        json.dump(state, f, indent=2)
    # Acknowledge in-flight messages so they don't re-deliver
    flush_inflight_messages()
    print("Checkpoint complete; exiting.")
    sys.exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)
signal.signal(signal.SIGINT, handle_sigterm)

Resume after restart

On next start, the agent reads /data/agent-state.json first, restores its working memory, and resumes. The transition from "stopped" to "running" is invisible to the operator — the agent picks up exactly where it left off. See the Persistent Workspaces guide for the full pattern.

Message deduplication patterns

Three layers where dedup happens:

For most production agent fleets, layers 1 + 3 (receiver-side cache + DB idempotency) is enough. Layers 2 and 4 are when those aren't sufficient.

Cost math

The architecture is essentially free; the model is the cost driver. At any scale, optimize the model layer first (cheaper model where quality permits, smaller prompts, prompt caching) before optimizing infrastructure.

Real-world patterns from the field

Pattern 1: Slack-DM-driven AP clerk

An accounting firm. Operator forwards an invoice to ap@firm.com. Email lands in Nina's inbox (Gmail forwarding). Nina's webhook receiver (a Lambda watching the Gmail Pub/Sub feed) pushes the email to her SQS queue. Nina (running on a ab0t.medium sandbox with auto-stop) wakes when SQS has a message; processes the invoice; posts the result to #ap-summary Slack via the Slack API; goes idle. End-to-end latency: 12-18 seconds. Cost per invoice: $0.08. Volume: 60-100/day.

Pattern 2: GitHub-PR-driven code reviewer

An engineering team. GitHub PR opened triggers a webhook. Receiver verifies HMAC, pushes to SQS. Code-reviewer agent (Cursor Cloud or Claude Code) pulls the message, spins up a fresh sandbox, checks out the branch, runs the test suite, drafts a review, posts as a PR comment, terminates the sandbox. Cost per PR: $0.05. Latency: 2-4 min including test run.

Pattern 3: Cron-fired competitive intel

A marketing team. EventBridge Schedule fires every weekday at 6am UTC. Lambda receiver pushes a "run-comp-sweep" message to SQS. The competitive-intel agent fan-outs to 30 sandboxes (one per competitor); each visits the competitor's site, screenshots, runs a visual diff against yesterday, returns findings. Aggregator combines, posts a Slack briefing in #competitive-intel. Cost per daily run: $0.45. End-to-end: 6 minutes.

Pattern 4: Calendar-aware month-end reconciliation

A bookkeeping firm. The agent reads its calendar daily. On the 26th of each month, it sees "month-end reconciliation due 28th." Schedules the work for the 27th. On the 27th, the cron-fired tick wakes the agent; it pulls last month's bank feeds, reconciles against the GL, drafts the month-end variance memo, posts to #monthly-close Slack channel. Cost per close: $4.20. Replaces 4-6 hours of bookkeeper time.

Pattern 5: Stripe-webhook-driven dunning

A SaaS startup. Stripe charge.failed webhook fires. Receiver verifies signature (Stripe rotates signing secrets — receiver checks all valid signatures). Pushes to SQS. Dunning agent reads customer record, checks past behavior (first failure? third?), drafts a personalized email, sends via Postmark, logs the dunning attempt in CRM. Cost per failure: $0.03. Recovery rate (vs no dunning): +18% revenue retained.

Anti-patterns: what not to do

• Polling loops in always-on sandboxes. Agent that polls every 5 seconds for new work burns compute 24/7. Use event-driven; auto-stop closes the gap.
• Webhook receiver doing the long work inline. Slack times out at 3 sec; if your agent takes 30 seconds, Slack retries and you process duplicates. Receiver pushes to queue; worker does the long work.
• Bare crontab on a single host. Host reboot at the cron tick = silent failure. Use cloud schedulers.
• No DLQ. Bad messages loop forever, fill the queue, drain budget. Always configure a DLQ with maxReceiveCount.
• Skipping HMAC verification "because we're behind a private VPC." Defense in depth. Verify HMAC even when network access is restricted; the next misconfiguration could expose the receiver.
• Same idempotency cache TTL as queue retention. If queue retention is 14 days but dedup cache is 24 hours, a message redelivered on day 2 looks new. Match the cache TTL to the queue retention or use FIFO with built-in dedup.
• Trusting timestamps from upstream. Clock skew between systems can be ±60 seconds. Add tolerance to timestamp-based logic.
• Re-using receiver code for unverified events. A receiver that handles authenticated webhooks should never accept events without verifying the signature, even if the source claims to be a different system.
• Fan-out at the receiver instead of the worker. The receiver should respond fast; don't spin up 50 sandboxes inline. Push 50 messages to the queue; let workers spin up at their own pace.
• EventBridge rules that match too broadly. A rule that matches all events from a partner bus and sends to a single target gets all your events; you wanted only the charge.failed ones. Use specific event-pattern matching.

Observability: what to monitor

For every event-driven agent fleet, set alerts on:

• Receiver 5xx rate. Spikes mean the receiver is failing; senders will retry, you'll see processing duplicates.
• Receiver latency. P99 should be under 1 second. Above that, you're approaching sender timeouts.
• Queue depth. Sustained growth means workers can't keep up; either scale workers or investigate why processing is slow.
• DLQ depth. Should be near-zero. Every message in DLQ is a failed run that needs triage.
• Cron tick fired metric. Missing fires mean the scheduler didn't trigger; alert on absence.
• Worker concurrency vs cap. If you're at 100% of cap and queue is growing, you're under-provisioned.
• Per-event end-to-end latency. Trace from receiver to result; track p50, p95, p99.
• Cost per event. Compute + model + queue. Spikes signal a runaway loop or a model regression.

Security considerations

• Verify every signature. Webhooks from external sources must be HMAC-verified. No exceptions.
• Restrict receiver ingress. Allow only known sender IP ranges (Slack, GitHub, Stripe publish theirs).
• Encrypt event payloads at rest. SQS and EventBridge support KMS-managed encryption; turn it on.
• Rotate signing secrets. Quarterly minimum; immediately on suspicion of compromise.
• Audit log every event. Sandbox Platform's audit log captures every wake, every action; supplement with your own application-level event log if compliance requires.
• Sandbox isolation per event. Per-task sandboxes mean a malicious payload can't affect other agents' work.
• Data residency. If you have residency requirements (EU GDPR, healthcare), use region-specific queues and sandboxes. EventBridge and SQS support cross-region replication if needed.

Comparison with workflow-orchestration platforms

Most production AI agent fleets we work with use Sandbox Platform's primitives directly because the agent harness (Claude Code, Devin, Cursor) is the unit they want to compose around — not a workflow-engine's DSL. If you're building from scratch and have no opinion on the harness, Inngest or Trigger.dev are reasonable starting points.

Frequently asked questions

What's the difference between webhook-driven and queue-driven agents?

Webhooks deliver events directly to your receiver; you process them inline. Queues (SQS, EventBridge, Redis Streams) buffer events between sender and receiver — your receiver pulls when ready. Use webhooks for low-latency human-facing flows (Slack mention, GitHub PR open). Use queues for high-throughput, retry-able, replayable workloads (overnight invoice processing, weekly competitive sweeps). Most production systems combine both: webhook lands the event, receiver pushes to a queue, agents pull from the queue.

How do I prevent the same event from being processed twice?

Idempotency keys. Each event carries a unique ID (the webhook delivery ID, the SQS message ID, your own UUID). Store processed IDs in a Redis or sqlite cache for 24 hours; reject duplicates. With SQS FIFO queues, you get exactly-once processing within a 5-minute window for free. Standard SQS queues are at-least-once and require your own dedup. Calendar-driven runs use the calendar event's ID + timestamp as the natural idempotency key.

What happens if my agent's sandbox is asleep when an event arrives?

The Sandbox Platform receiver wakes the sandbox automatically. If the sandbox is stopped, the platform resumes it (8-15 sec) before delivering the event. If the sandbox is running but idle, the event arrives in the agent's inbox immediately. Auto-stop on idle costs nothing in latency for event-driven agents — the wake delay is invisible to humans on the other end of the workflow.

Should I use SQS or EventBridge?

SQS is a queue. EventBridge is a router. Use SQS when you have one producer and one consumer. Use EventBridge when you have one producer and many consumers (or many producers fanning into one consumer). For most AI agent workloads, SQS is enough. EventBridge is the right choice when you want a single agent fleet to react to events from multiple SaaS systems via partner-event integrations (Stripe, Zendesk, Datadog, GitHub all publish to EventBridge directly).

How do I handle webhook secret rotation without downtime?

Configure the receiver to accept two valid secrets simultaneously (current + previous). When you rotate, push the new secret to the receiver first; the receiver accepts both. Then update the sending side (Slack, GitHub, Stripe) to sign with the new secret. After 24 hours of clean traffic on the new secret, retire the old one.

What about cron jobs that don't fire?

The number-one production failure for cron-driven agents. Set an expected-fire-time alert that pages you if the job hasn't started by N minutes after its scheduled time. Use cloud schedulers (AWS EventBridge Schedules, Google Cloud Scheduler) with built-in retry instead of bare crontab on a single host. Audit log each fire so you can prove the job actually ran.

How do I scale to thousands of events per second?

Three layers. (1) Receiver: API Gateway + Lambda for webhooks (handles 10K req/sec without thinking); for higher, ALB + cloud containers. (2) Queue: SQS standard at 3000 msg/sec per queue, or partition into N queues. (3) Worker: agent fan-out per the patterns in the Running 100 in Parallel guide. Cap concurrency at the slowest downstream system. The bottleneck is rarely the queue; it's the model API rate limit or the third-party portal rate limit.

Can long-running agents survive sandbox restarts?

Yes — by design. Sandbox Platform's auto-stop preserves disk; agents serialize state to /data/agent-state.json before stop and restore on resume. Heartbeats during long jobs let you detect a crashed agent vs an idle one. SIGTERM handlers give the agent 30 seconds to checkpoint cleanly before forced stop. See Persistent Workspaces for the full pattern.

What's the right SQS visibility timeout for agent workers?

Set it to 1.5× your worst-case task duration. If a typical task takes 60 seconds and the worst-case is 5 minutes, set visibility timeout to 7.5 minutes. Too short: messages re-deliver before the agent finishes, causing duplicate work. Too long: a crashed agent's task sits invisible for the duration, delaying retry. SQS supports per-message visibility extension (ChangeMessageVisibility) so your agent can extend if it knows the work will run long.

How do I test webhook receivers locally?

Use ngrok or Cloudflare Tunnel to expose your local receiver to the public internet, then point the sending side (GitHub, Slack, Stripe) at the tunneled URL. Each major sender has a "replay this event" button in their dashboard for re-running failed deliveries while you debug. Better: use the platform's webhook-replay feature, which re-fires any past event into your receiver from the audit log.

What is exactly-once processing and is it real?

Exactly-once means a message is processed exactly one time, no more, no less. In practice it requires either (a) the receiver is idempotent (processing the same message twice yields the same result), or (b) the queue + receiver have transactional semantics (SQS FIFO + worker-side dedup window). Pure exactly-once across distributed systems is theoretically impossible (the Two Generals problem); what production systems achieve is "at-least-once with idempotent receivers," which is functionally exactly-once for practical purposes.

How does this compare to Inngest or Trigger.dev?

Inngest and Trigger.dev are workflow-orchestration platforms — you write the workflow as code; they handle the events, retries, and observability. They're great if you want all-in-one. Sandbox Platform's event integration is more primitive — we ship the receiver patterns, the queue integration, the calendar / cron support, and you compose them with your own agent harness (Claude Code, Codex CLI, Devin). Use Inngest / Trigger.dev when you want fewer pieces; use the platform's primitives when you want more control.

Do I need an event-bus at all, or can the agent just poll?

Polling works for low-frequency / low-urgency triggers (check the inbox every 5 minutes). It breaks down at three places: latency (humans expect Slack-mention responses in seconds, not minutes), cost (always-on polling burns sandbox compute), and reliability (the polling agent is a single point of failure). Event-driven scales to zero between events and wakes on demand. For anything user-facing or with bursty arrival patterns, event-driven wins.

How do I unit-test the receiver code?

Capture real signed payloads from the sender's test mode (Slack's Block Kit Builder, GitHub's webhook tester, Stripe's CLI). Replay them against your receiver in tests. For HMAC verification logic, generate signed test payloads in the test setup using the same algorithm. The platform ships a test-events CLI that replays a fixture set against your receiver.

What about events from systems that don't have webhooks?

Three patterns: (1) wrapper service that polls the source on a schedule and emits a webhook; (2) the source's API change-feed (some systems expose change-feeds via Pub/Sub or SSE); (3) a CDC tool (Debezium, Fivetran, AirByte) that emits events from database changes. Pick by what the source supports.

What's next