How to Use Karpenter to Right-Size GPU Nodes on Amazon EKS

With fixed node groups, you choose GPU instance types in advance and pay for them whether they're busy or not. Karpenter works the other way round: it watches for pods that can't be scheduled, launches instances that fit them and removes nodes that are no longer needed.

Step 1: Install Karpenter

Follow the EKS getting-started guide for your Karpenter version. It covers the controller, IAM roles and the interruption queue for Spot capacity. Karpenter's APIs changed at version 1.0, so check that any example you copy matches your version.

Step 2: Create a GPU NodePool

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["g5", "g6"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
      taints:
        - key: nvidia.com/gpu
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu
  limits:
    nvidia.com/gpu: 8
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 10m

The limits block caps how many GPUs this pool can create, which protects your budget.

Step 3: Configure the node class

The EC2NodeClass selects the AMI, subnets and security groups. Use an EKS-optimized accelerated AMI so NVIDIA drivers are already installed, and give nodes enough disk for large images.

Step 4: Target the pool

Inference pods request nvidia.com/gpu and tolerate the taint. Karpenter picks an instance type from the allowed families that satisfies the request.

Step 5: Decide where Spot fits

Training and batch jobs that checkpoint can run on Spot capacity at a large discount. Latency-sensitive inference usually stays on-demand. Separate NodePools make that choice explicit.

Things to watch

• Consolidation and running models. WhenEmpty removes only idle nodes. More aggressive consolidation can move running model servers, so add Pod Disruption Budgets.
• Capacity shortages. GPU instances are sometimes unavailable in a zone. Allow several instance families and zones.
• Startup time includes launching the instance, pulling the image and loading the model. Keep warm capacity for services that can't wait.
• Budget alerts. NodePool limits cap the number of GPUs, but they don't tell anyone when the cap is reached. Pair them with a cost alert so a runaway job gets noticed the same day.

Start with one NodePool for a single GPU workload and review its costs after a week.