Benefits of Auto-Stopping Idle GPU Environments

Friday, 6 pm: someone closes their laptop and forgets the GPU notebook. Monday, 9 am: it has been billing for 63 hours. This article covers the benefits of auto-stopping idle GPU environments, the dev boxes and notebooks your sprint team spins up, and gives you the script that does it. The examples use AWS, but the rule works anywhere you can stop a VM.

The auto-stop rule for idle GPU environments

The rule: if every GPU stays under 5% utilization for 60 minutes, stop the instance. Disks survive, so nobody loses work. This script runs on the box itself, reads utilization from nvidia-smi, and stops the instance through the EC2 API. It assumes the AWS CLI is installed.

#!/usr/bin/env bash
# /usr/local/bin/gpu-autostop.sh
set -euo pipefail

IDLE_MINUTES="${IDLE_MINUTES:-60}"
UTIL_THRESHOLD="${UTIL_THRESHOLD:-5}"   # percent
STATE_FILE=/run/gpu-idle-since          # /run is cleared on every boot

# Peak utilization of any GPU over about 10 seconds
peak=0
for _ in $(seq 10); do
  util=$(nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits | tr -d ' ' | sort -n | tail -1)
  if [ "$util" -gt "$peak" ]; then peak="$util"; fi
  sleep 1
done

if [ "$peak" -ge "$UTIL_THRESHOLD" ]; then
  rm -f "$STATE_FILE"   # busy: reset the idle clock
  exit 0
fi

now=$(date +%s)
[ -f "$STATE_FILE" ] || echo "$now" > "$STATE_FILE"
idle_seconds=$(( now - $(cat "$STATE_FILE") ))
[ "$idle_seconds" -ge $(( IDLE_MINUTES * 60 )) ] || exit 0

# IMDSv2: fetch a token, then the instance ID and Region
token=$(curl -sf -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
meta() { curl -sf -H "X-aws-ec2-metadata-token: $token" "http://169.254.169.254/latest/meta-data/$1"; }
instance_id=$(meta instance-id)
region=$(meta placement/region)

logger -t gpu-autostop "GPU idle for ${idle_seconds}s, stopping ${instance_id}"
aws ec2 stop-instances --instance-ids "$instance_id" --region "$region"

Each run samples ten seconds of peak utilization, so a short burst resets the clock. The timer file lives in /run on purpose: it is cleared on boot, so a box you start on Monday does not stop itself over a stale Friday timestamp. The metadata calls follow the EC2 instance metadata docs.

Run it every five minutes from cron, so the box stops about an hour after the GPU goes quiet:

# /etc/cron.d/gpu-autostop
PATH=/usr/local/bin:/usr/bin:/bin
IDLE_MINUTES=60
UTIL_THRESHOLD=5
*/5 * * * * root /usr/local/bin/gpu-autostop.sh

Then give the instance role permission to stop only instances that opted in with a tag. The condition style follows the EC2 example IAM policies.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "ec2:StopInstances",
      "Resource": "arn:aws:ec2:<region>:<your-account-id>:instance/*",
      "Condition": {
        "StringEquals": { "aws:ResourceTag/auto-stop": "true" }
      }
    }
  ]
}

Opt a box in and label it for reporting:

aws ec2 create-tags --resources <instance-id> \
  --tags Key=auto-stop,Value=true Key=sprint,Value=s42 Key=owner,Value=<your-name>

Benefits of auto-stopping idle GPU environments

1. Direct hourly savings. A g5.xlarge (one NVIDIA A10G) lists at $1.006 per hour on demand in us-east-1, so check your own Region. Friday 6 pm to Monday 9 am is 63 hours, about $63 for one forgotten box. With the rule it bills roughly one extra hour, about $1. Eight engineers forgetting one box each is around $500 for a single weekend, and each idle weeknight (6 pm to 9 am) adds about $15 per box. A stopped instance stops incurring usage charges, though its EBS volumes keep billing.

2. A cost report per sprint that explains itself. Because every box carries sprint and owner tags, one query shows what the sprint's GPUs cost and who ran them:

aws ce get-cost-and-usage \
  --time-period Start=2026-10-05,End=2026-10-19 \
  --granularity DAILY \
  --metrics UnblendedCost \
  --filter file://gpu-filter.json \
  --group-by Type=TAG,Key=owner

{
  "Tags": {
    "Key": "sprint",
    "Values": ["s42"],
    "MatchOptions": ["EQUALS"]
  }
}

Activate sprint and owner as cost allocation tags first; AWS says it can take up to 24 hours for a tag key to appear and up to 24 more to activate. A flat weekend in the daily numbers is your proof the rule works. Stopping through the API also leaves a CloudTrail record, so "who stopped my notebook?" has an answer.

3. Shared GPUs stay available. Most sprint teams share a few boxes or a fixed budget. A forgotten box ties one up, and the teammate who needs a GPU on Monday waits or launches another. Auto-stop returns the capacity on its own, and getting a box back is one command: aws ec2 start-instances --instance-ids <instance-id>.

4. A rule people can repeat. "GPU under 5% for an hour, the box stops, the disk stays." That fits in a working agreement instead of a policy document. Make exceptions visible with auto-stop=false, show the sprint cost line at review, and ask in the retro who got caught and whether the threshold needs tuning.

When auto-stop is not worth it

Long jobs with quiet GPUs. Data loading, CPU-side preprocessing, checkpoint writes and waiting on an external API can all drop utilization under 5%, and the rule will stop the job mid-run. Set those boxes to auto-stop=false: the IAM condition no longer matches, so AWS refuses the stop call and the box keeps running.

Data on instance store. Stopping erases RAM, instance store volumes and the auto-assigned public IPv4 address. Keep work on EBS or S3, and use an Elastic IP if you need a fixed address, noting that Elastic IPs bill while the instance is stopped.

Tiny fleets. Two people sharing one box that is busy all day will spend more time on this script than it saves. The payoff scales with how many boxes get forgotten.

Dev-environment waste is the easiest GPU spend to cut before it reaches production budgets, and one rule in your working agreement does most of it.