Spot / preemptible instances in real workloads
Where spot and preemptible capacity actually works, how to handle interruptions cleanly, and the realistic savings magnitude for different workload types.
Spot (AWS) and preemptible (GCP) instances can cut compute cost by 60–90 % compared with on-demand. They can also vanish with two minutes of notice (or less). The difference between a successful spot strategy and a series of 3 a.m. pages is whether the workload was designed for interruption from the start.
I only put two categories of work on spot: (1) truly fault-tolerant, horizontally scalable batch or worker fleets, and (2) mixed fleets where a baseline of on-demand or reserved capacity absorbs the critical path and spot fills the elastic portion. Everything else stays on more predictable capacity.
One data-processing pipeline I moved to a mixed fleet (40 % on-demand + 60 % spot) saved roughly 55 % on the compute line while keeping the same end-to-end latency SLO. The key was checkpointing every few minutes and making the workers completely stateless with respect to the local disk. When interruptions occurred the queue simply re-delivered the work; no human was involved.
Workloads that survive interruption cleanly
- Stateless workers that pull from a queue and acknowledge only after success
- Batch jobs that checkpoint progress and can resume
- CI runners, rendering farms, data-processing pipelines with retries
- Kubernetes node pools that can be drained and rescheduled
Workloads that fight interruption:
- Single-instance stateful services without fast failover
- Long-running in-memory computations without checkpointing
- Anything that assumes a fixed IP or a multi-hour uninterrupted lease
The first question I ask is “what happens if this machine disappears in 90 seconds?” If the answer is “we lose work or page someone,” it does not belong on spot.
Handling the interruption signal
Every major cloud gives a termination notice. The application or the node agent must listen for it and start a clean shutdown.
# AWS example – poll the instance metadata for the termination time
TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \
http://169.254.169.254/latest/meta-data/spot/instance-actionOn receipt the process should:
- Stop accepting new work
- Finish or checkpoint current work
- Deregister from load balancers / service discovery
- Exit cleanly so the orchestrator can replace it
For Kubernetes the node-termination handler or the cloud controller does most of this; the application still needs to respect SIGTERM and finish within the grace period.
Mixed fleets and the realistic savings
A pure-spot fleet for a latency-sensitive user-facing service is usually a bad idea. A mixed fleet (for example 30–50 % on-demand/reserved baseline + spot for the remainder) captures most of the savings while keeping a reliable floor. The exact ratio depends on how spiky the load is and how quickly new spot capacity can be acquired in the chosen pools.
I measure the actual interruption rate and the time-to-replace for the chosen instance families and availability zones. Those two numbers determine whether the savings survive contact with reality.
Practical limits and gotchas
- Spot capacity is not evenly distributed; some families and AZs are chronically tight.
- Price can spike; set a max price or use capacity-optimized allocation strategies.
- Persistent disks and Elastic IPs still cost money while the instance is gone.
- Licensing and software that is tied to a specific host can become complicated.
When the interruption rate or the capacity availability makes the operational overhead higher than the savings, I move that workload back to on-demand or reserved and keep spot for the parts that remain tolerant.
Spot is a powerful cost lever when the workload is designed for it. It is an operational liability when it is treated as “cheap on-demand.” Design for interruption first, then measure the real interruption rate and the real savings. Only then decide how large a fraction of the fleet can live on the interruptible capacity.
I also keep a simple dashboard: current spot percentage, interruption count per day, average time-to-replace, and the resulting cost versus pure on-demand. When any of those numbers drifts, the allocation strategy or the instance families are reviewed. Spot savings are real, but they are not automatic; they require the same operational attention as any other production capacity.
