How to Schedule GPU Workloads on Kubernetes Without Wasting Cards

A GPU node can cost more per hour than a rack of CPU nodes, and Kubernetes won't stop you wasting it. Pods that don't need a GPU land on GPU nodes, and pods that need a fraction of a card hold an entire one.

Step 1: Install the device plugin

Kubernetes doesn't see GPUs by default. The NVIDIA device plugin, usually installed through the NVIDIA GPU Operator, advertises cards as the nvidia.com/gpu resource. Managed services like EKS, GKE and AKS offer GPU node images that handle much of this setup.

Step 2: Keep other pods off GPU nodes

Taint GPU nodes so only pods that ask for GPUs are scheduled there:

kubectl taint nodes gpu-node-1 nvidia.com/gpu=present:NoSchedule

GPU pods add a matching toleration. Without the taint, ordinary workloads fill the node and GPU pods no longer fit.

Step 3: Request GPUs explicitly

resources:
  limits:
    nvidia.com/gpu: 1

By default, GPUs are allocated as whole units, and a pod that asks for one card gets a whole card to itself.

Step 4: Share cards where it fits

For small workloads, such as a lightweight embedding model or development notebooks:

• Time-slicing lets several pods take turns on one GPU. It's simple, but pods can affect each other's memory and speed.
• Multi-Instance GPU (MIG) splits supported cards, such as the A100 and H100, into isolated instances with dedicated memory.

Use MIG where production workloads need isolation, and time-slicing for development.

Step 5: Measure utilization

Deploy NVIDIA's DCGM exporter to send GPU metrics to Prometheus, then look at utilization and memory per pod over a week. Cards that stay below 20 percent utilization are candidates for sharing or consolidation.

Step 6: Scale nodes with demand

Use the cluster autoscaler or Karpenter so GPU nodes are added when pods are pending and removed when they're empty. GPU nodes left idle overnight are one of the most common sources of waste.

Things to watch

• Driver and CUDA versions must match what your containers expect. Pin them and upgrade deliberately.
• Large model images take minutes to pull onto a new node. Allow for that when autoscaling.
• Batch jobs and latency-sensitive inference compete badly on the same cards. Keep them apart.

Check the utilization metrics first. They usually show exactly where the money goes.