The Canonical Benchmark For True-Cold Agent Compute.
Most sandbox benchmarks measure how fast a provider can unpause a stale memory snapshot or hand out a pre-warmed container. ColdBench measures the real systems problem: booting an isolated microVM from nothing to verified guest execution, then tearing it down cleanly.
Our Benchmarks Are The Only Snapshots We Use
ColdBench times one thing: a create request to a microVM verified ready, on a true-cold boot every time. No snapshots, no warm pools, no reuse.
Request → verified ready, by microVM size
600 launches per size · linear scale from 0 · left is faster
| microVM size | Fastest | p50 | p90 | p99 |
|---|---|---|---|---|
| 1 vCPU / 4 GiB | 51 ms | 55 ms | 56 ms | 60 ms |
| 2 vCPU / 8 GiB | 51 ms | 56 ms | 60 ms | 67 ms |
| 2 vCPU / 16 GiB | 52 ms | 59 ms | 64 ms | 69 ms |
| 4 vCPU / 32 GiB | 58 ms | 67 ms | 70 ms | 72 ms |
600 true-cold launches per size, one at a time, on Azure Standard_F8amds_v7 (centralus). Runs of Sep 11, 2026. Times are measured on the host, from create request to verified ready.
As of 09-25-2026, Setsuna ColdBench has clocked 241,488 launches across 34 Azure VM types, every one verified, unique, and cleaned up. A cumulative total, not the sample charted here.
How ColdBench Times Every Launch
A valid ColdBench run must pass all four verification gates without warm pools, cached VM state, or background pre-allocation.
Request Arrival (t = 0 ms)
The clock starts on the host the instant the create request arrives—before allocation, kernel boot, or device setup.
True-Cold Kernel Boot
Every launch boots a fresh Linux kernel from zero. No memory snapshots, no pre-warmed pools, and no recycled state.
Verified Guest Ready
The clock stops only after the microVM is reachable and a verification probe inside the guest confirms readiness.
Complete Teardown
Every benchmarked microVM is verified unique and torn down completely—compute, disk, and network attachment—before the run concludes.
Setsuna ColdBench vs. Published Sandbox Figures
Setsuna figures are host-measured request → verified ready (p50 of 600 true-cold launches on Azure Standard_F8amds_v7, 2 vCPU / 16 GiB). Competitor figures are published client-side create or cold-start times, so measured spans differ and are not like-for-like.
- Setsuna 59 ms
True-cold custom runtime
Host request → verified guest ready (p50 of 600)
- E2B 717 ms
Firecracker snapshot resume [2]
Client-side sandbox create [1]
- Vercel Sandbox 1,852 ms
Firecracker microVM [1]
Client-side sandbox create [1]
- Fly Machines ~2,000+ ms
Firecracker cold start / suspend [3]
Published cold start for common apps [3]
- Modal 2,437 ms
gVisor container sandbox [1]
Client-side sandbox create [1]
- [1] LogRocket — Comparing AI agent sandbox platforms: E2B, Modal, Daytona, and more, Aug 4, 2026. Measured create: E2B 717 ms, Vercel Sandbox 1,852 ms, Modal 2,437 ms.
- [2] E2B — Infrastructure architecture, Retrieved Aug 2026. Firecracker microVMs with snapshot resume.
- [3] Fly.io — Machine suspend and resume, Retrieved Aug 2026. “~2+ seconds” cold start for common apps.