Unevictable Pods

Unevictable pods in Kubernetes are pods that cannot be removed from a node during eviction processes due to scheduling constraints, resource requirements, or disruption policies. They can block node scale-down, cause resource fragmentation, and increase infrastructure costs, making them an important factor in maintaining efficient and flexible cluster operations.

Unevictable pods are pods that cannot be safely removed from a node by Kubernetes during eviction processes, such as node draining, resource pressure handling, or cluster autoscaler scale-down.

These pods prevent Kubernetes from reclaiming or redistributing resources, often blocking node termination and reducing overall cluster efficiency.

What does eviction mean in Kubernetes

Eviction is the process by which Kubernetes removes pods from a node to maintain cluster health or optimize resource usage.

Evictions typically occur during:

• Resource pressure: such as memory or disk shortages
• Node maintenance: draining a node for upgrades or repairs
• Cluster autoscaler scale-down: removing underutilized nodes

Evicted pods are usually rescheduled onto other nodes, assuming suitable capacity exists.

Why pods become unevictable

A pod may become unevictable when Kubernetes cannot safely terminate and reschedule it. Common causes include:

• Pod Disruption Budgets (PDBs): Prevent eviction if it would reduce the number of available replicas below a defined threshold
• No valid placement options: The pod cannot be scheduled on any other node due to:
  • high resource requests
  • strict node affinity or selectors
  • taints and tolerations
• Local storage usage: Pods using node-local storage may not be safely rescheduled elsewhere
• High priority workloads: Critical pods may be protected from eviction
• Strict scheduling constraints: Hard rules that limit where a pod can run

Operational impact

Unevictable pods can have significant effects on cluster efficiency:

• Blocked scale-down: nodes cannot be removed even if underutilized
• Resource fragmentation: available capacity exists but cannot be consolidated
• Higher infrastructure cost: unused nodes continue running
• Reduced flexibility: harder to rebalance workloads across the cluster

How to identify unevictable pods

You can detect unevictable pods through:

• Cluster autoscaler logs: indicating scale-down failures
• kubectl describe pod: showing scheduling constraints and resource requests
• Pod Disruption Budget status: checking if eviction is allowed
• Scheduling events: indicating no suitable node for relocation

How to fix or avoid unevictable pods

To reduce the likelihood of unevictable pods:

• Adjust Pod Disruption Budgets to allow safe evictions
• Relax overly strict affinity or placement rules
• Right-size resource requests to improve schedulability
• Minimize reliance on node-local storage
• Review workload constraints and placement policies

Example

A cluster autoscaler attempts to remove an underutilized node, but one pod cannot be evicted.

The pod has a strict node affinity and high resource requests, making it impossible to place on other nodes. As a result, the node remains active, leading to unnecessary infrastructure cost.

Final thoughts

Unevictable pods are workloads that cannot be safely rescheduled during eviction processes, often due to constraints or resource limitations. They can block node scale-down, reduce cluster efficiency, and increase costs. Identifying and addressing these constraints is key to maintaining a flexible and cost-effective Kubernetes environment.