Out of Storage? How to Efficiently Resize PersistentVolumes

By Omer Hamerman
Principal DevOps Engineer

In Kubernetes, running out of storage can disrupt your workloads, lead to application downtime, or force you into time-consuming manual interventions like migrating data or recreating volumes. Thankfully, Kubernetes offers a practical solution: PersistentVolume resizing, which lets you expand your storage dynamically to keep your applications running smoothly.

Whether you’re managing growing databases or unexpected spikes in storage needs, resizing is an essential feature that helps prevent downtime and eliminates the need for complex, time-consuming manual storage migrations. Let’s walk through how it works, when to use it, and how to do it the right way.

Why Resize Volumes? It’s All About Keeping Up

Applications grow. Databases expand. Log files pile up. If your PersistentVolume (PV) isn’t big enough, you risk hitting capacity limits that can slow down or even break your application. Resizing lets you expand a volume dynamically without recreating it—saving you the hassle of manual migration or downtime.

What You Need to Know Before You Start

Resizing volumes requires careful preparation to avoid disruptions. If your setup isn’t configured correctly—like missing volume expansion support or incompatible storage classes—You may risk leaving your application stuck with insufficient storage. To ensure a successful expansion, follow these steps:

1. Check Your StorageClass: Resizing only works if the StorageClass used by your PersistentVolumeClaim (PVC) has allowVolumeExpansion set to true. Here’s what it looks like:

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: expandable-storage
   provisioner: kubernetes.io/aws-ebs
   allowVolumeExpansion: true

2. Dynamic Provisioning Is Key: Your PVC must be dynamically provisioned (created using a StorageClass). Static volumes won’t work with resizing.
3. Your Storage Provider Matters: Make sure your Container Storage Interface (CSI) driver supports resizing. Most major providers like AWS EBS, Google PD, and Azure Disk do.
4. Kubernetes Version: Volume expansion became a stable feature in Kubernetes 1.19. If you’re on an older version, it’s time to upgrade.

Step-by-Step: How to Resize a PersistentVolume

Ready to resize your volume? Follow these simple steps:

1. Start with the StorageClass

First, confirm that your StorageClass has allowVolumeExpansion: true. You can check it with:

kubectl get storageclass expandable-storage -o yaml

2. Update the PVC

Edit the spec.resources.requests.storage field in your PVC manifest to the desired size. For example, to increase a PVC from 10Gi to 20Gi:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

Apply the updated PVC:

kubectl apply -f pvc.yaml

3. Verify the Resize

Once you’ve updated the PVC, Kubernetes will handle resizing the associated PersistentVolume. Check the status with:

kubectl get pvc example-pvc

You should see the new size reflected under the CAPACITY column.

Online vs. Offline Resizing: What’s the Difference?

Not all storage providers handle resizing the same way:

• Online Resizing: Supported by most modern providers (e.g., AWS EBS, Azure Disk), this lets you resize the volume while the pod is running. No downtime required!
• Offline Resizing: Some providers require the pod to be stopped before resizing can occur. This can mean scheduling downtime for your application.

Always check your storage provider’s documentation to know what’s supported.

Can You Automate Volume Resizing?

Kubernetes doesn’t have built-in automation for resizing PersistentVolumes yet. But there are ways to make the process smoother:

1. Monitoring Tools:

Tools like Prometheus can monitor storage usage and alert you when you’re approaching capacity. Pair this with a script to automatically update your PVC.

Example monitoring script logic if storage_usage >= 80%:
new_size = current_size * 1.5 # Increase by 50%
update_pvc_size(new_size)

This monitoring script will:

• Monitor storage usage with Prometheus
• Trigger when capacity reaches 80% (a safe buffer before hitting limits)
• Automatically increase the PVC size by 50% to provide good headroom while avoiding too-frequent resizing

This can be implemented either through a custom operator or a monitoring-triggered script.

2. Custom Operators:

For building a Kubernetes operator to handle PVC usage monitoring and volume resizing, Kubebuilder is an excellent framework to get started. Here’s a helpful starting point:

• The official Kubebuilder Book provides comprehensive documentation and tutorials for creating operators.

It covers essential concepts like:

• Setting up your operator project
• Defining Custom Resources (CRs)
• Implementing controllers
• Testing and deployment strategies

You can use Kubebuilder to create a controller that:

• Monitors PVC usage metrics
• Implements threshold-based resizing logic
• Handles the volume expansion process automatically

3. Cloud Features:

Some cloud providers offer alternatives like Amazon EFS (Elastic File System) that can automatically scale storage outside of Kubernetes. While not directly integrated, these can complement your cluster’s storage strategy when used with the EFS CSI driver.

However, be aware of these trade-downs:

• Performance: EFS typically has higher latency compared to block storage options like EBS
• Cost: EFS can be more expensive for certain workload patterns due to its pricing model based on storage and throughput
• Setup Complexity: Requires proper configuration of the EFS CSI driver and appropriate IAM roles/permissions
• Network Dependencies: EFS access depends on network connectivity and proper VPC configuration

Zesty Kompass:

Zesty Kompass is a cloud cost optimization platform for Kubernetes. One of Zesty Kompass’ main features offers an innovative solution to Kubernetes storage challenges through intelligent volume management. Key features include:

• Automated volume scaling based on real-time cluster needs
• Up to 70% cost savings through smart provisioning
• Zero-downtime operation with virtual filesystem technology
• Continuous monitoring and optimization without disrupting running pods

Real-World Scenarios: When Should You Resize?

• Growing Databases: Your database is growing faster than expected, and you need more storage to keep it running smoothly.
• Bursty Applications: Applications like logging systems can see sudden spikes in storage usage. Resizing helps you respond quickly.
• Preventive Scaling: Stay ahead of storage demands by monitoring usage and resizing before you hit capacity limits.

Best Practices for Resizing Storage Volumes

Resizing is powerful, but it’s not something to approach haphazardly. Here’s how to get it right:

1. Test in Staging: Before resizing in production, test the process in a non-critical environment to ensure compatibility.
2. Enable Monitoring: Use tools like Prometheus or Grafana to keep an eye on storage usage and predict when resizing might be needed.
3. Plan for Downtime (if needed): If your storage provider requires offline resizing, schedule it during low-traffic periods to minimize impact.
4. Document the Process: Keep a clear record of resizing steps and storage configurations for your team.

Wrapping It Up: Storage Resizing Made Simple

Expanding PersistentVolumes in Kubernetes can feel complex initially because it involves understanding specific prerequisites, like storage class configurations and provider capabilities. However, with the proper setup, the process becomes straightforward and efficient. By enabling volume expansion, understanding your storage provider’s capabilities, and following best practices, you can keep your applications running smoothly no matter how fast your storage needs grow.