Why your Kubernetes scheduler can't handle AI workloads
Imagine this scenario: You have a distributed training job with 16 worker pods, each requesting 1 GPU. 4 GPUs are currently available. The default Kubernetes scheduler (kube-scheduler) may schedule those 4 pods while the remaining 12 stay pending. Meanwhile, those 4 GPUs are reserved by pods that…
Imagine this scenario: You have a distributed training job with 16 worker pods, each requesting 1 GPU. 4 GPUs are currently available. The default Kubernetes scheduler (kube-scheduler) may schedule those 4 pods while the remaining 12 stay pending. Meanwhile, those 4 GPUs are reserved by pods that cannot make progress until the full distributed job is ready. No one else on the cluster can use them. Your job isn't actually training. The other cluster users' jobs can't use those reserved GPUs either. The cluster is busy doing absolutely nothing. This is a classic partial-scheduling deadlock. By default, kube-scheduler doesn't gang-schedule; the feature exists in alpha but ships disabled. It schedules pods one at a time, meaning it has no concept of "all pods in this job must start together, or none of them start." The second problem is that kube-scheduler lacks multi-node fabric topology awareness. While it can track basic on-node constraints such as available CPU or memory capacity, it doesn’t account for which separate nodes share the same NVIDIA Quantum, or how high-bandwidth interconnects are routed across the cluster network. It places pods wherever capacity exists. For distributed training, this introduces severe communication latency that throttles performance at scale. For teams running small jobs on a few GPUs, this inefficiency is tolerable. For AI/ML organizations running distributed training across dozens or hundreds of GPUs, it breaks the entire workflow.Source: Lambda Labs — Published — Category: Models