Custom Resources (Kubernetes)

Custom Resources in Kubernetes are user-defined objects that extend the Kubernetes API, allowing teams to define and manage their own configuration or domain-specific objects just like built-in ones (e.g., Pods, Services).

Background

Kubernetes was designed with extensibility in mind. While it comes with a set of built-in resource types (like Deployments, ConfigMaps, etc.), these aren’t always enough for real-world needs. That’s where Custom Resources (CRs) come in.

Introduced as part of Kubernetes’ API extension mechanism, CRs let you define new kinds of objects without modifying Kubernetes source code or recompiling anything.

They’re powered by the CustomResourceDefinition (CRD), a Kubernetes-native way to register new resource types.

Why Custom Resources Matter

Custom Resources allow you to:

• Introduce new abstractions tailored to your system (e.g., Database, BackupPolicy, MLJob)
• Declaratively manage complex applications using the same tooling you already use for Kubernetes
• Build Operators—automated controllers that manage the lifecycle of custom objects

This enables platform teams, SREs, and DevOps engineers to codify operational knowledge into Kubernetes-native workflows.

Key Concepts

1. CustomResourceDefinition (CRD)

A CRD is what tells Kubernetes to recognize a new resource type. When you apply a CRD YAML, Kubernetes adds a new API endpoint, such as:

/apis/mygroup.example.com/v1alpha1/myresources

From then on, users can create and manage MyResource objects just like Pods or Services.

2. Custom Resource (CR)

Once the CRD is registered, a Custom Resource is an instance of that type. For example:

apiVersion: apps.example.com/v1
kind: WebApp
metadata:
  name: my-app
spec:
  image: nginx
  replicas: 3

This defines a new WebApp object that your custom controller or operator can act on.

3. Controller / Operator

A controller watches for changes to your Custom Resource and takes action—such as creating deployments, scaling workloads, or updating configurations. When that controller implements complex logic, it’s often called an Operator.

Use Case Examples

Example 1: Define a Database resource

apiVersion: infra.mycompany.com/v1
kind: Database
metadata:
  name: customer-db
spec:
  engine: postgres
  version: "14"
  storage: 20Gi

This could be backed by an operator that provisions a PostgreSQL instance, attaches volumes, and ensures backups.

Example 2: ML Pipeline Step

apiVersion: ai.org/v1beta1
kind: TrainingJob
metadata:
  name: train-bert
spec:
  model: bert
  dataset: s3://mybucket/data
  gpu: true

A controller watches for TrainingJob resources and spins up GPU workloads accordingly.

Custom Resources vs. Built-in Resources

How to Create a Custom Resource

1. Create a CRD (example)

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: webapps.apps.example.com
spec:
  group: apps.example.com
  names:
    kind: WebApp
    plural: webapps
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                image:
                  type: string
                replicas:
                  type: integer

2. Apply the CRD

kubectl apply -f webapp-crd.yaml

3. Create your Custom Resource

apiVersion: apps.example.com/v1
kind: WebApp
metadata:
  name: frontend
spec:
  image: nginx:latest
  replicas: 2

Apply it with kubectl apply -f frontend.yaml.

Use Cases in Real-World Platforms

Challenges and Considerations

• Validation: You must define schemas in CRDs to prevent bad input
• RBAC Complexity: Managing access to new resources can get tricky
• Controller Maintenance: You’re responsible for writing and maintaining custom controllers
• Versioning: You need to handle version upgrades just like with any Kubernetes resource

Despite these, CRDs are the de facto method for extending Kubernetes.

Resources and Further Reading

• Kubernetes Docs – Extending the Kubernetes API with CRDs
• Kubebuilder – Framework for building Kubernetes APIs using Go
• Operator SDK – Toolkit to build Operators
• Kubernetes API Conventions