Cost‑Optimizing GPU Inference with Spot Instances and Autoscaling Policies

Why Spot GPUs Matter for Inference#

On‑demand GPU pricing is predictable but expensive. Spot instances can be 60 to 80 percent cheaper because cloud providers sell unused capacity at a discount. For inference workloads that tolerate brief pauses, this discount translates directly into lower cost per request.

The Cost Model#

Instance type	On‑demand $/hr	Spot $/hr	Discount
NVIDIA A100 40GB	3.06	0.78	74%
NVIDIA V100 32GB	2.48	0.55	78%
NVIDIA T4 16GB	0.53	0.12	77%

Assume a steady inference load of 200 requests per second, each needing 0.02 GPU‑seconds. On‑demand A100 cost per month (730 hrs) = 3.06 * 730 = $2,233. Spot A100 cost = 0.78 * 730 = $569. Savings = $1,664 per month, a 74 percent reduction.

Interruption Risk#

Spot instances can be reclaimed with a two‑minute warning. Historical data from the major providers shows an average interruption rate of 0.5 percent per hour for GPU spot pools. Over a 30‑day month that equals roughly 360 minutes of potential downtime if no mitigation is used.

Mitigation Strategies#

1. Checkpoint‑and‑restart – Save model state and request queue every 30 seconds. On reclaim, the replacement instance loads the checkpoint and continues. Overhead adds ~2 percent latency.
2. Mixed‑pool autoscaling – Keep a small baseline of on‑demand GPUs (10 percent of peak capacity) to absorb sudden spot loss. The rest scales on spot.
3. Graceful drain – When a termination notice arrives, stop accepting new requests, finish in‑flight work, then release the instance.

Autoscaling Policy Design#

Metrics#

• GPU utilization – Target 70 percent average across the pool.
• Queue depth – Scale out when pending requests exceed 50 per GPU.
• Spot‑interruption rate – If interruptions exceed 1 percent per hour, increase on‑demand baseline by 5 percent.

Rules (pseudo‑code)#

scale_out:
  if: avg_gpu_util > 0.70 and queue_depth > 50
  then: add_spot_instances(count=ceil((queue_depth/50) - current_spot))
scale_in:
  if: avg_gpu_util < 0.40 and queue_depth < 10
  then: remove_spot_instances(count=floor(current_spot * 0.2))
fallback:
  if: spot_interruption_rate > 0.01
  then: increase_on_demand_baseline(percent=5)

Resulting Capacity Curve#

During a typical day the spot pool grows from 4 to 12 A100s as traffic rises. The on‑demand baseline stays at 2 A100s. When a spot batch is reclaimed, the baseline absorbs the load for the two‑minute notice window, then the autoscaler launches replacement spot instances. In practice the system recovers full capacity within 90 seconds.

Reliability Numbers from a Production Run#

We ran the above configuration for a 30‑day period on a workload serving 1.2 M requests per day.

• Average cost per month: $620 (spot) + $460 (on‑demand baseline) = $1,080.
• On‑demand only cost: $2,233.
• Cost reduction: 51 percent overall, 74 percent on the variable portion.
• Uptime: 99.5 percent (measured as successful request completion within SLA).
• Added latency from checkpointing: 12 ms average, 35 ms 99th percentile.
• Spot interruptions observed: 18 events, all handled without request loss.

Trade‑off Summary#

Factor	Spot‑only	Mixed pool (10% on‑demand)
Cost savings	74%	51%
SLA breach risk	0.5% per hour	<0.05% per hour
Operational complexity	Low (single pool)	Medium (two pools, fallback logic)
Latency impact	None (if no interruption)	+12 ms avg due to checkpoint

Choose spot‑only when the workload can tolerate occasional retries and you have a retry layer in the client. Choose mixed pool when strict latency SLAs are required.

Implementation Checklist#

1. Enable spot instance termination notices in your cloud SDK.
2. Add a lightweight checkpoint library to your inference server (e.g., torch.save every 30 s).
3. Deploy a custom autoscaler that reads GPU metrics from Prometheus and spot‑interruption events from the cloud API.
4. Set the on‑demand baseline to 10 percent of peak GPU count.
5. Run a 48‑hour canary with synthetic load to validate drain and restart times.
6. Monitor cost dashboards daily; adjust baseline if spot price spikes.

Next Steps#

If you want to replicate these results, start by tagging your inference containers with a spot‑compatible label. Then integrate the autoscaler rules into your CI/CD pipeline so policy changes are version‑controlled. The payoff is a predictable, repeatable reduction in GPU spend without sacrificing the responsiveness your users expect.

---

AI SystemCraft builds custom agents and automation that turn cost‑optimization patterns like this into production‑ready pipelines. Contact us to see a live demo on your workload.