Benefits of Kubernetes Network Policies for AI Workloads

Your inference pod has a GPU, a copy of your model weights, and by default, a clear path to every other pod in the cluster. That last part is the problem. Kubernetes ships wide open: any pod can talk to any pod unless something says otherwise. For a stateless web app that's an inconvenience. For a cluster running model training, RAG pipelines, and customer-facing inference side by side, it's an open floor plan for anyone who gets a foothold.

Kubernetes network policies for AI workloads fix this by making the network topology match your actual trust boundaries instead of your namespace's default "everyone can reach everyone." Here's what that buys you.

Contains a compromised inference pod

If an attacker gets remote code execution through a prompt injection that reaches a tool-calling agent, or through a vulnerable dependency in your serving stack, the first thing they do is scan the local network for something more valuable. A default-deny policy on your inference namespace means that scan comes back empty.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: inference
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This single policy, applied to the inference namespace, drops all traffic in and out of every pod there until you explicitly allow it. Everything below is about punching the specific holes you actually need.

Stops silent model and data exfiltration

A pod that can reach the open internet can also upload your model weights, embeddings, or customer prompts to an attacker's server, and nothing in the Kubernetes dashboard will flag it as unusual. Egress policies close that path by only letting inference pods talk to the specific endpoints they need, like your object store or model registry.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-model-store
  namespace: inference
spec:
  podSelector:
    matchLabels:
      app: llm-server
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.20.0.0/16
    ports:
    - protocol: TCP
      port: 443

On a CNI that supports it, like Calico, Cilium, or GKE's Dataplane V2, you can go further and allow egress by DNS name (objectstore.internal.svc) instead of a raw CIDR, which survives IP churn in a way ipBlock rules don't.

Separates training from inference traffic

Training jobs and inference services have different risk profiles. Training pods often mount raw, unfiltered datasets; inference pods are internet-facing. Putting them in separate namespaces with policies that deny cross-namespace traffic by default means a bug in your inference gateway can't accidentally become a path into your training data lake, and vice versa.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-gateway-only
  namespace: inference
spec:
  podSelector:
    matchLabels:
      app: llm-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: gateway
    ports:
    - protocol: TCP
      port: 8080

Protects the vector database from lateral movement

Your vector store or feature store is often the least protected, most valuable thing in the cluster: it has your customers' embedded documents, often without the access controls that a traditional database gets. Scoping ingress on the vector DB pods to only the RAG service that queries them means a compromised sidecar in an unrelated app can't just port-scan its way to your embeddings.

Gives you an actual audit trail

With a policy-aware CNI like Cilium, denied connections show up in flow logs you can ship to your SIEM. That turns "we think nothing unusual happened" into "here's the list of every connection attempt our inference namespace rejected last week," which is the kind of evidence a security review or SOC 2 audit actually wants.

When it's not worth the effort

If you're running a single-tenant cluster with no external-facing services, or you've already got a service mesh enforcing mutual TLS and authorization between every workload, layering network policies on top adds YAML to maintain for security you already have. And they're only as good as your CNI: kubenet and older Flannel setups don't enforce NetworkPolicy at all, so check your plugin before you write a single rule.

Network policies won't stop a determined attacker who's already inside an allowed path, but they turn "one compromised pod means the whole cluster is up for grabs" into a contained incident, which is the difference between a bad afternoon and a breach notification.