The EBS PV Availability Zone Trap During EKS Upgrades

1. The Incident: Infinite Pending State

During an EKS cluster upgrade from 1.32 to 1.34, we initiated a rolling update of our node groups. Our core stateful service, a Zookeeper cluster backing our transactional matching engine, was terminated to allow the pods to be rescheduled on the new 1.34 nodes.

The nodes rolled, but the Zookeeper pods never recovered. They remained in the Pending state indefinitely. Inspecting the pod events showed:

0/141 nodes are available:
131 node(s) didn't match Pod's node affinity/selector
8 node(s) didn't match PersistentVolume's node affinity
2 node(s) had untolerated taint(s)

The critical clue was: "node(s) didn't match PersistentVolume's node affinity". Our stateful containers could not be scheduled because the available nodes were incompatible with the volumes containing their persistent data.

2. The Root Cause: gp2 PV Availability Zone Binding

To understand why this happened, we inspected the age of the Zookeeper Persistent Volumes:

NAME              CLAIM               STORAGECLASS   AGE
data-zookeeper-0  pvc-14c17617-...    gp2            6y75d
data-zookeeper-1  pvc-329dd18d-...    gp2            6y75d
data-zookeeper-2  pvc-56613520-...    gp2            6y75d

These volumes were AWS EBS gp2 volumes created over six years ago. When an EBS volume is created in AWS, it is physically tied to a single Availability Zone (e.g., ap-northeast-2a). Because of this physical constraint, Kubernetes automatically attaches a nodeAffinity rule to the PV, restricting scheduling to that specific AZ:

kubectl get pv {pv-name} -o yaml | grep -A 5 nodeAffinity
# topology.kubernetes.io/zone: ap-northeast-2a

During the EKS upgrade, our Terraform configuration provisioned new worker nodes for the target node group. However, due to ASG (Auto Scaling Group) balancing, the new nodes were spun up in ap-northeast-2b and ap-northeast-2c.

This created an impossible scheduling constraint:

1. The pod's nodeSelector required the node to belong to the designated matching-engine node group.
2. The PV's nodeAffinity required the node to reside in ap-northeast-2a.
3. No new nodes in the matching-engine node group existed in ap-northeast-2a.

As a result, the scheduler was completely blocked, and the pods remained Pending.

3. The Blast Radius: Cascading Failures

Because Zookeeper was down, our core matching engine could not boot. And because the matching engine was down, dependent API gateways, transaction processors, and web applications failed their readiness probes. Over 20 backend microservices went offline, causing rolling updates to freeze and extending what should have been a 30-minute node rotation into a 5-hour outage.

4. Best Practices for Stateful EKS Upgrades

To prevent this scheduling deadlock in future upgrades, we established a strict pre-flight audit checklist:

A. Audit PV Availability Zones

Before starting a cluster upgrade, identify all volumes that have been in service for a long period and check their AZ bindings:

kubectl get pv -A -o custom-columns='NAME:.metadata.name,CLAIM:.spec.claimRef.name,SC:.spec.storageClassName'

B. Pin Auto Scaling Groups to the Correct Subnets

For node groups hosting stateful workloads, ensure that the Terraform definition forces node creation in the subnets matching the PV AZs:

module "stateful_node_group" {
  source             = "terraform-aws-modules/eks/aws//modules/self-managed-node-group"
  subnet_ids         = [data.aws_subnet.ap_northeast_2a.id] # Pin to the correct AZ
}

C. Use Multi-AZ-aware Storage (EFS) or WaitForFirstConsumer

While migrating legacy gp2 to gp3 is recommended for performance and cost, gp3 is still an EBS volume and remains physically bound to a single AZ. To avoid AZ locks, use multi-AZ-aware storage like AWS EFS, or configure your StorageClass with volumeBindingMode: WaitForFirstConsumer. This delays volume binding until the pod is scheduled, ensuring the pod and volume are provisioned in the same AZ.