Kubernetes doesn't schedule GPUs like it schedules CPUs

If you ask Kubernetes for cpu: 500m you get half a core, throttled by cgroups, sharing a physical CPU with whatever else landed on the node. If you ask for nvidia.com/gpu: 1 you get an entire physical GPU, exclusively, whether your model needs 2GB or 40GB of it.

This is not an oversight. It follows from what the scheduler knows how to do.

Compressible versus non-compressible#

Kubernetes divides resources into two classes.

CPU is compressible. If a container exceeds its limit, the kernel throttles it. The container slows down; it does not die. Time-slicing a core between processes is something operating systems have done for fifty years, and the cgroup CFS quota mechanism makes it a scheduling parameter.

Memory is non-compressible. You cannot give a process "less memory, slower". Exceed the limit and the OOM killer resolves it. So the scheduler treats memory as a hard reservation.

GPUs are non-compressible and the kernel has no native abstraction for dividing them. There is no cgroup controller for CUDA cores. From the scheduler's point of view a GPU is an opaque device that a container either has or does not.

Extended resources and device plugins#

Because the kubelet has no built-in concept of a GPU, NVIDIA hardware is surfaced through the device plugin API. The plugin runs as a DaemonSet, discovers devices on the node, and advertises them to the kubelet:

apiVersion: v1
kind: Node
status:
  capacity:
    cpu: "64"
    memory: 264057456Ki
    nvidia.com/gpu: "8"   # advertised by the plugin, not discovered by kubelet
  allocatable:
    nvidia.com/gpu: "8"

Extended resources have deliberate constraints. They must be integers — there is no nvidia.com/gpu: 500m. And requests must equal limits, so there is no burstable class:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: inference
      image: registry.internal/serving:2026.4
      resources:
        limits:
          # No request field needed — for extended resources
          # Kubernetes sets request = limit automatically.
          nvidia.com/gpu: 1

Everything confusing about GPU scheduling falls out of those two rules.

Why your utilisation is bad#

A typical inference deployment runs a model that needs 6GB of VRAM on an A100 with 80GB. Kubernetes gives that pod the whole card. Seven eighths of the device is unreachable — not idle-and-available, but genuinely unschedulable, because the scheduler has already decremented allocatable to zero for that device.

Multiply across a fleet and the numbers get ugly fast. On one cluster I looked at, average GPU memory utilisation was 19% while nvidia.com/gpu allocation sat at 100%. Every card was "full" and almost every card was empty.

The three ways out#

Time-slicing. Configure the device plugin to advertise one physical GPU as several logical ones. Contexts are swapped on the device; there is no memory isolation and no fairness guarantee.

version: v1
sharing:
  timeSlicing:
    resources:
      - name: nvidia.com/gpu
        replicas: 4   # one A100 now advertises as 4 allocatable GPUs

Cheap to turn on, works on any card. But a pod that allocates 40GB will OOM its neighbours, because nothing enforces the split. Fine for development and notebooks, risky for production serving.

MIG. On A100/H100, Multi-Instance GPU partitions the card in hardware into up to seven instances with dedicated memory, cache, and SM slices. Real isolation, advertised as distinct resource names:

resources:
  limits:
    nvidia.com/mig-1g.10gb: 1

The catch is that the partition layout is set at the node level and changing it drains the node. You are picking a fixed shape for your workload mix in advance.

MPS. The Multi-Process Service multiplexes CUDA contexts through a single daemon, giving better throughput than naive time-slicing with some memory limits. It sits between the other two on both isolation and flexibility.

Choosing#

The decision follows from your workload mix. Uniform models of a known size: MIG, sized to the model. Heterogeneous and unpredictable: MPS, and accept the weaker isolation. Untrusted tenants sharing a cluster: MIG, and do not let anyone talk you out of it.

What you should not do is leave whole-device allocation in place and buy more cards. That is the expensive way to solve a scheduling problem.