How to Enforce Policies on Kubernetes AI Workloads With Kyverno

Platform teams write rules for AI workloads: always set resource requests, never run as root, use approved registries only, keep GPU workloads in approved namespaces. Under deadline pressure, those rules get skipped. Kyverno is a Kubernetes policy engine that checks resources as they're created and can reject the ones that break the rules.

Step 1: Install Kyverno

Install it with its Helm chart into its own namespace. It runs as an admission controller, so every create and update request passes through it.

Step 2: Start with a policy in audit mode

This policy requires images to come from your approved registry:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: approved-registries
spec:
  rules:
    - name: registry-check
      match:
        any:
          - resources:
              kinds: ["Pod"]
      validate:
        failureAction: Audit
        message: "Images must come from registry.acme.com."
        pattern:
          spec:
            containers:
              - image: "registry.acme.com/*"

In Audit mode, violations are reported but not blocked, and Kyverno writes policy reports you can review. Older Kyverno versions set the failure action for the whole policy instead of per rule, so match the syntax to your version.

Step 3: Add the policies that matter for AI workloads

• Resource requests required, so GPU and memory-hungry pods can't be scheduled without them.
• No privileged containers or root users, which also closes off many container escape paths.
• GPU requests only in approved namespaces, so expensive cards aren't used by accident.
• Signed images only, using Kyverno's image verification rules.
• Required labels such as team and cost center, so costs can be attributed.

Kyverno's policy library includes ready-made versions of many of these.

Step 4: Review reports, then enforce

Run in audit mode for a couple of weeks. Fix existing violations, add justified exceptions, then switch policies to Enforce one at a time.

Step 5: Test policies like code

Keep policies in Git and test them in CI with the Kyverno CLI against sample manifests, so a policy change can't block every deployment by mistake.

Things to watch

• System namespaces. Exclude namespaces such as kube-system from most policies, or you may block cluster components.
• Exceptions. Use policy exceptions with an owner and an expiry date, rather than weakening a policy for everyone.
• Availability. If Kyverno is down and configured to fail closed, deployments stop. Run it with several replicas.

Turn on the registry and resource-request policies in audit mode today, and review the reports at the end of the week.