How to avoid costly instance selection mistakes in Karpenter

By Ido Slonimsky

Tech Lead, Zesty

Your Karpenter configuration can make or break your cluster’s efficiency. Configure it poorly and you risk locking yourself into expensive nodes that never consolidate.

I have seen clusters where Karpenter was installed correctly, yet costs increased, and the root cause was almost always the same: NodePool design.

The key is to configure it widely enough to give it the breadth to make the best decisions for you. This guide provides steps to design NodePools that give Karpenter enough flexibility to optimize while protecting you from costly mistakes.

First, Understand What Karpenter Is Optimizing

Before tuning anything, clarify how Karpenter works.

What Karpenter Actually Does

Karpenter watches for unschedulable pods and provisions nodes that satisfy:

• Pod resource requests
• Node affinity and anti-affinity
• Taints and tolerations
• Topology constraints
• NodePool requirements

It chooses from available AWS instance types that match your NodePool constraints. It also continuously evaluates whether nodes can be consolidated to reduce cost.

Consolidation works by:

1. Attempting to reschedule pods from one node to others.
2. Terminating the empty or underutilized node if possible.

AWS provides hundreds of instance type combinations across families, sizes, architectures, and pricing models. If you restrict Karpenter too tightly, you reduce its search space. If you leave it completely open, you may get instance types that do not align with your financial or operational goals.

Prerequisites

Before applying the steps in this guide, ensure:

• You are running Karpenter v1 or later.
• IAM roles and instance profiles are configured for Karpenter.
• You have permissions to create or modify NodePools.
• Consolidation is enabled in your Karpenter configuration.
• You have visibility into cost data through AWS Cost Explorer or similar.

Checkpoint:

kubectl get nodepools

You should see your active NodePools.

Now that we understand the decision model, let’s design the constraint matrix properly.

Treat NodePools as a Constraint Matrix

A NodePool defines the boundaries of what Karpenter is allowed to provision.

It is effectively a matrix that may include:

• Instance families
• Instance sizes
• Architecture
• Capacity type such as spot or on-demand
• Availability zones
• Custom requirements

Here is a simplified example:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-workloads
spec:
  template:
    spec:
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["m6i", "m6a", "c6i", "c6a"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge", "2xlarge"]
        - key: karpenter.k8s.aws/capacity-type
          operator: In
          values: ["spot"]

This definition gives Karpenter several instance families and multiple sizes. That flexibility is critical for cost optimization.

According to AWS documentation, broader instance diversification improves spot availability and price stability. AWS recommends diversified capacity pools to reduce interruption risk. The same principle benefits Karpenter’s optimization logic.

Action Step: Broaden Intelligently

Review your existing NodePools:

• Are you limiting to a single instance family?
• Are you using only one size?
• Are you restricting to a single availability zone?

If yes, expand gradually.

Checkpoint:

After applying changes:

kubectl apply -f nodepool.yaml
kubectl describe nodepool general-workloads

Verify the requirements section reflects multiple families and sizes.

Avoid the Silent Killer: Over-Restricting Instance Types

Many teams unintentionally restrict Karpenter too much.

Common examples:

• Only allowing one instance family because it was used historically.
• Hardcoding a single size such as m6i.2xlarge.
• Avoiding spot entirely out of caution.

This reduces Karpenter’s ability to:

• Bin-pack workloads efficiently.
• Replace nodes with cheaper alternatives.
• Consolidate across a diverse fleet.

If a pod requests 2 vCPU and 4 GiB, there are dozens of valid instance types. If you only allow one, you force Karpenter into suboptimal choices.

Action Step: Expand Instance Diversity

Instead of:

values: ["m6i"]

Use:

values: ["m6i", "m6a", "m5", "c6i", "c6a"]

Ensure the families are appropriate for your workload profile. For general workloads, mixing compute and memory balanced families increases scheduling flexibility.

Expected Outcome:

• New nodes may come from different families.
• Spot interruption rates may decrease.
• Consolidation events should increase over time.

Monitor:

kubectl logs -n kube-system deploy/karpenter

Look for consolidation and provisioning events.

Now let’s address the opposite risk, which is equally dangerous.

Overpaying for CPU and memory you don’t use?
Rightsize pods to reduce spend without risking performance.

Learn how Zesty automatically aligns resources with real-time demand while preventing CPU throttling and OOM issues.

Read More

Add Guardrails Before Large Instances Burn Your Budget

Allowing every possible instance type can lead to expensive mistakes.

The Large Node Trap

Consider a 128 vCPU instance. If a single non-evictable pod lands on it, Karpenter may be unable to consolidate that node later.

You end up paying for an oversized node hosting a tiny workload.

This is a common cost trap.

The Bare Metal Surprise

Bare metal instances are significantly more expensive. If not explicitly excluded, Karpenter may consider them valid.

Action Step: Cap Maximum Size

Limit instance sizes to a reasonable upper bound:

- key: karpenter.k8s.aws/instance-size
  operator: In
  values: ["large", "xlarge", "2xlarge", "4xlarge"]

Avoid including extreme sizes unless required.

To exclude specific categories:

- key: karpenter.k8s.aws/instance-category
  operator: NotIn
  values: ["metal"]

Checkpoint:

After deployment, confirm no oversized nodes are launched:

kubectl get nodes -o wide

Validate instance types via:

kubectl get nodes -L node.kubernetes.io/instance-type

With safe guardrails in place, we can design for resilience without sacrificing cost control.

Design Tiered NodePools for Resilience and Cost Discipline

Workloads are not limited to a single NodePool. A pod can match multiple pools if requirements align.

This allows a tiered strategy.

Primary NodePool

• Spot
• Savings Plan aligned
• Broad but controlled instance diversity

Secondary Fallback NodePool

• On-demand
• Slightly broader compatibility
• Still capped at reasonable sizes

Example fallback:

- key: karpenter.k8s.aws/capacity-type
  operator: In
  values: ["on-demand"]

This ensures that if spot capacity is unavailable or blocked by anti-affinity rules, workloads still schedule.

Expected Behavior:

• Most nodes come from the primary pool.
• Fallback activates only under constraints.
• No runaway provisioning of extreme instance types.

Monitor distribution:

kubectl get nodes -L karpenter.sh/nodepool

You should see most nodes tied to the primary pool.

Rightsize Pods to Unlock Consolidation

Karpenter can only optimize based on pod requests.

If your pods request 4 vCPU but use 0.5 vCPU, you are blocking consolidation.

Kubernetes scheduling is request-driven, not usage-driven.

According to Kubernetes documentation, resource requests determine bin-packing decisions. Overstated requests directly reduce scheduling density.

Action Step: Audit Resource Requests

Identify overprovisioned workloads:

kubectl top pods --all-namespaces

Compare usage versus requests:

kubectl get pod <pod-name> -o yaml | grep resources -A 5

Adjust deployment specs:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

Expected Outcome:

• More pods fit per node.
• Consolidation events increase.
• Node count gradually decreases.

If you use automated rightsizing tools, ensure changes are applied continuously. Karpenter benefits from smaller, accurate requests.

Verify That Consolidation Is Actually Working

Do not assume savings. Verify them.

Check Consolidation Activity

kubectl logs -n kube-system deploy/karpenter | grep Consolidation

You should see nodes being evaluated and occasionally terminated.

Check Cost Trends

Use AWS Cost Explorer:

• Filter by EC2.
• Compare before and after NodePool changes.
• Monitor instance family distribution.

Healthy Signals

• Diverse instance types in use.
• Gradual node count reduction under steady workload.
• No persistent oversized nodes with minimal utilization.

If consolidation rarely occurs, investigate:

• Overprovisioned pods.
• Pod disruption budgets blocking eviction.
• Anti-affinity rules preventing rescheduling.

This final section ties everything together.

How Zesty Strengthens Your Karpenter Strategy

Everything we covered so far assumes your pod requests reflect real usage. In most clusters, they do not. Teams overprovision CPU and memory to stay on the safe side, and Kubernetes schedules strictly on those requests. When pods ask for more than they use, Karpenter is forced to provision larger nodes and consolidation becomes harder.

Zesty continuously rightsizes pod requests based on actual usage patterns. With accurate requests, more pods fit per node, consolidation succeeds more often, and large underutilized instances are easier to remove. If you are investing in well-designed NodePools, automated rightsizing ensures Karpenter can fully capitalize on that design.

Putting It All Into Practice

To avoid costly instance selection mistakes in Karpenter:

1. Give Karpenter broad instance diversity.
2. Exclude extreme or unnecessary instance categories.
3. Cap maximum instance sizes.
4. Implement tiered NodePools.
5. Continuously rightsize pods.
6. Monitor consolidation and cost trends.

Cost optimization in Kubernetes is iterative. You refine constraints, observe behavior, and adjust. When NodePools are thoughtfully designed and workloads are properly sized, Karpenter becomes a reliable cost optimization engine instead of a hidden cost amplifier.

Next steps:

• Experiment in a staging cluster.
• Introduce spot capacity gradually.
• Review PodDisruptionBudgets for consolidation blockers.
• Track savings over 30 day intervals to capture meaningful trends.

With disciplined NodePool design and rightsized workloads, you can safely give Karpenter the flexibility it needs to do its job well.