How to improve system resiliency against traffic spikes

By Avishay Shamay

Backend Team Lead

Traffic spikes are a headache every Kubernetes (K8s) practitioner dreads – from clogging up your pod deployment to throwing off your scheduling, messing with your resource allocation, or even knocking your services offline. Knowing how disruptive these issues can be, we created a guide of practical strategies to help you manage those spikes effectively. By fine-tuning your Kubernetes setup, you can boost system responsiveness and build a more resilient environment without stretching your budget. Stick with us as we walk you through these methods and show you how to keep your system performing smoothly, no matter what comes its way.

What Exactly Happens to Our Kubernetes Workload During a Spike

When the Wave Hits: An Initial Surge

• The Onset: Imagine this — your RPS shoots up out of nowhere. Immediately, your pods start feeling the pressure, gobbling up CPU and memory resources, which might just push them to their breaking point.

HPA to the Rescue

• HPA in Action: The Horizontal Pod Autoscaler (HPA) isn’t just sitting back. It’s constantly monitoring your resources and when things get too hot to handle, it starts deploying more pods to distribute the load. But remember, it typically checks every 15 seconds by default, which might delay its reaction time.

From Zero to Hero: Pod Lifecycle Dynamics

• Pods in Waiting: Triggered by the HPA, new pods pop up to manage the load. But if there’s no room at the inn, they stay in an ‘unschedulable’ limbo until new nodes are ready.
• Node Provisioning Drama: This is where tools like Cluster Autoscaler or Karpenter come into play, spinning up new nodes to house your waiting pods. This step is crucial and can be a major bottleneck.
• The Deployment Dance: Got nodes? Great! Now, the Kubernetes scheduler gets these pods placed, and the node’s kubelet pulls the necessary images.
• Application Warm-up: With images ready, your application starts booting up. This is crunch time, where startup probes play referee to decide when your app is really ready to jump into the fray.

Stability Regained

• Balancing Act: As these new pods come online and start sharing the load, your system begins to stabilize, returning to its usual efficient self.

Turbocharging Your Kubernetes Response Time

Making Your HPA Snappier

• Quick Checks for Quick Response: Shorten those HPA metric check intervals. Yes, it adds a bit of strain on your metrics server, but the speed gain can be worth it.

Node Provisioning: Need for Speed

• Choosing Your Champion: Not all autoscalers are created equal. Put Cluster Autoscaler and Karpenter to the test across different scenarios to see which one tops the speed charts for your needs.
• Fast Track Your Node Startup: Lean on AMIs like Bottlerocket or optimized EKS AMIs for quicker node readiness.

Image Management: Slim and Trim

• Cut Down the Fat: Embrace multi-stage builds and scratch base images to minimize image size — less to pull means quicker starts.
• Registry Efficiency: Opt for the fastest registry available to ensure swift image pulls.

Application Boot: Every Second Counts

• Slash Startup Times: Know the tricks to get your apps up and running in no time. Tools like AOT compilation for Java can sidestep typical delays.
• Embed, Don’t Load: Pack those static assets into your image to avoid post-startup downloads.

Startup Probes: Tuned to Perfection

• Probe Settings Matter: A tweak in the periodSeconds of your startup probes can dramatically cut down the time till your pods are battle-ready.

Locking Down Resilience

Setting up your Kubernetes to smoothly handle traffic spikes is about being one step ahead. With these tweaks and optimizations, your infrastructure won’t just survive the next big surge—it’ll thrive. Keep an eye on the impacts, adjust as necessary, and watch your Kubernetes environment become a fortress of reliability, ready for anything the digital world throws its way.

What is the Impact of Traffic Spikes

In Kubernetes, a traffic spike can mean a few things but commonly refers to a sudden and large increase in requests per second (RPS) hitting one or more applications in the cluster. A surge like that can push the scalability and resilience of services to the limit, potentially harming their performance and availability. Other forms of traffic spikes can be an increase in RPS in a specific endpoint that is very resource intensive, a message queue blowing up, etc.

Lifecycle of a Workload During a Spike

Initial Surge in Requests: The spike begins with a sudden rise in RPS, which immediately increases the loads of the pods in the cluster. If it intensifies, the pods might exhaust their CPU and memory, compromising the system’s performance.

Horizontal Pod Autoscaler (HPA) Kicking In: To manage the increased load, the HPA attempts to increase the number of pods based on predefined thresholds. By default, the HPA monitors the resource utilization every 15 seconds, which might cause a delay in detecting an incoming surge. As traffic fluctuates, HPA adjusts the pods deployment to ensure effective load distribution across the infrastructure.

Pod Lifecycle from Creation to Readiness:

1. Create pods: When the HPA is triggered, it initiates the creation of new pods to handle the additional load. If no immediate resources are available, pods remain in a ‘unschedulable’ state, as they wait for nodes to become available to host them.
2. Provision nodes: If no immediate resources are available, the cluster’s node autoscaler (like Cluster Autoscaler or Karpenter) activates, provisioning new nodes to accommodate the pending pods. Nodepool scaling is a crucial step that can cause significant delays.
3. Scheduling and Deployment: Once nodes are available, the Kubernetes scheduler assigns the new pods to these nodes. Then, the kubelet on each node starts pulling the necessary images.
4. Application Start-up: After the container images are pulled, the code within the containers of the pods begin to initialize. During this phase, startup probes ensure that the application is ready to handle requests.

Once the pod has passed these checks, it is finally marked as ‘Ready’ and begins to receive traffic.

As new pods become ready and start serving traffic, the load on the other pod normalizes, and the system’s state stabilizes.

Strategies for Improving Scale Speed

Below are some effective strategies to avoid scaling delays and ensure swift transitions in cases of traffic spikes:

Optimize Horizontal Pod Autoscaler (HPA) Settings:

• Make the HPA More Responsive: Shortening the intervals at which the HPA checks metrics can help the system react faster to changes in demand. However, increasing the frequency of these checks will put an extra strain on the metrics server, so it should be done with consideration.

Enhance Node Provisioning Speed:

• Weigh Your Options: Compare the effectiveness of Cluster Autoscaler and Karpenter on different applications to identify the best tool for each workload.
• Speed Up Node Startup: Using fast booting AMIs, like Bottlerocket or those optimized EKS AMIs, can really speed things up and get those nodes online quicker.

Optimize Image Management:

• Shrink Image Size: Reduce the image size using techniques like multi-stage builds with scratch base images. Smaller images mean faster pull times.
• Pick the Right Registry: It also helps to choose faster registries and make sure we’re pulling images as efficiently as possible.

Application Boot Optimization:

• Cutting Down Startup Time: There are several tricks to reduce how long our apps take to start. Every second counts. For example, Java can be slow because of JIT compilation. Using AOT compilation with tools like Quarkus can help dodge these delays.
• Pre-load Assets: Embed static assets right into the image. This will avoid the need to download them when the app starts, which is a common delay.

Configuring Startup Probes:

• Fine-Tuning Startup Probes: Adjusting the periodSeconds in startup probes can help. If we reduce this setting, our pods are considered ready sooner, which means they can start handling traffic faster.

Continuous Resilience

Setting your Kubernetes system up to effectively handle traffic spikes takes proactive measures and strategic adjustments. By optimizing HPA settings, enhancing node provisioning speeds, and honing application startup processes, your infrastructure will become more resilient to unexpected conditions. Implement these changes, monitor their impact, and adjust as needed. This will make your Kubernetes environment ready for whatever comes its way.