How to Build a Kubernetes Operator: A Practical Developer's Guide Print

Kubernetes already knows how to keep a Deployment or a Service in the state you asked for. A Kubernetes Operator extends that same behavior to things Kubernetes was never built to understand natively — an external database, a message queue, a piece of infrastructure in another cloud entirely. This guide walks through what an operator actually is, how the pieces underneath it fit together, and how to build a small one end to end.

Part 1: What Is a Kubernetes Operator?

Kubernetes constantly compares what you asked for against what's actually running, and a controller is the piece of code that closes the gap between the two. The built-in Deployment controller is a good example: if you delete a Pod it manages, the controller notices and replaces it.

That comparison-and-correction cycle is called reconciliation: look up the desired state, look up the actual state, decide what to do, and repeat. Critically, a reconciler recomputes that decision fresh every single time it runs — it never assumes it caught every event that happened in between. That's what makes it resilient: even if a run gets skipped or arrives late, the next run still gets the right answer.

An operator applies that exact loop to a resource type Kubernetes doesn't know about out of the box. You define a Custom Resource to describe the desired state of whatever you're managing, and you write a controller that knows how to reconcile it. Everything else — informers, workqueues, finalizers, status fields — exists purely to make that one loop dependable.

Operator vs. Controller vs. CRD

These three terms get used interchangeably, but they're three different layers of the same system:

• CRD (CustomResourceDefinition): a schema registered with the Kubernetes API server so it knows a new resource type exists. On its own, a CRD does nothing — it just gives the API server something to validate and store.
• Controller: any process running a reconciliation loop against a resource type — this includes Kubernetes' own built-in controllers, not just custom ones.
• Operator: a controller (or small group of them) built around a custom resource, with enough domain knowledge baked in to manage that resource's entire lifecycle — provisioning, recovery, upgrades — without a human in the loop.

Every operator is a controller, but not every controller is an operator, and a CRD with no controller behind it is just an unused schema.

Why Not a Helm Chart, a CronJob, or a Script?

Each of the obvious alternatives falls short in the same place: none of them keep watching.

• A Helm chart renders YAML and applies it once. If something it created later drifts or gets deleted, Helm has no idea until someone runs helm upgrade by hand.
• A CronJob gives you a loop, but at the cost of freshness — state can be stale for up to a full interval, and nothing is carried between runs.
• A one-off script only reacts when triggered (manually or by CI) and does nothing about drift the rest of the time. It's also rarely written with retries and idempotency as a first-class concern.

An operator is event-driven and continuous instead: the API server tells it the instant something changes, and it keeps reconciling for that resource's entire lifetime. That's worth the extra cost — its own RBAC, its own failure modes, its own observability surface — specifically when the problem is keeping something continuously correct, not just rendering some YAML once. If rendering YAML once really is the whole job, a Helm chart remains the right tool.

Part 2: The Machinery Behind the Loop

Custom Resource: Spec vs. Status

Every Custom Resource carries the identity fields any Kubernetes object has (kind, name, namespace, labels), plus two fields that are entirely yours to define: spec and status. This split isn't cosmetic — it maps directly onto the reconciliation loop.

• Spec is desired state. Whoever creates or edits the resource writes it; the controller only ever reads it.
• Status is observed state. Only the controller writes it, to record what it found and what it did.

A client writing directly to status is working around the controller instead of through it — which is why status is usually served as its own subresource with separate permissions, something we'll set up shortly.

Watching for Change: Informers, Listers, Workqueues

A reconciler doesn't poll the API server asking "did anything change yet?" — three pieces work together to avoid that:

• An informer opens a long-lived watch against the API server and keeps a local, in-memory cache of every object of a given type.
• A lister reads from that cache instead of the API server, so checking "does this resource exist?" is a local lookup, not a network call.
• A workqueue sits between the informer and the reconciler. When something changes, the informer enqueues a key (namespace + name) rather than the object itself. The queue deduplicates and rate-limits on your behalf — ten rapid updates to the same object collapse into one pending item.

This is also why a reconciler receives a key, not an object — by the time a worker pulls that key off the queue, the object may already have changed again. The reconciler always looks up current state itself rather than trusting whatever triggered it.

The Manager and Leader Election

The manager is the process that owns the shared cache, the client used to read and write objects, and the health/readiness/metrics endpoints the rest of the cluster relies on. Every reconciler you register runs inside one manager.

If you run more than one replica for availability, you don't want two of them reconciling the same object simultaneously. The manager solves this with leader election: replicas compete for a lease, exactly one holds it and actively reconciles, and the rest sit idle until that lease stops being renewed.

The Reconciliation Loop, in Detail

A reconciler's entry point is called with only a namespace and a name — no diff, no event payload. It has to fetch the object itself, compare spec against what it can observe, and decide what to do. That constraint shapes everything below.

Idempotency. Because a reconciler can be called any number of times for the same object — back to back, out of order, or after a long gap — it has to reach the same end result regardless. A reconciler that blindly calls create every run breaks the moment it runs twice. The fix: always check current state first — create only if missing, update only if different, delete only if it shouldn't exist.

Event-driven, plus a resync. A reconcile fires on a watch event for the resource itself, and by convention on anything it owns. Most controllers also set a periodic resync so the loop runs on a schedule even with zero watch events — necessary because state can drift for reasons no watch would ever catch.

Requeues. Sometimes a single pass can't finish because whatever it's waiting on is still converging elsewhere. A reconciler can ask to be called again after a delay without treating that as a failure — this is how it polls something slow rather than blocking inside one call.

Error handling. Returning an error also requeues, but with exponential backoff instead of a fixed delay, so a persistently failing reconcile doesn't hammer whatever it's failing against. It's worth distinguishing errors worth retrying (a timeout, a lock conflict) from ones that aren't (a spec that will never be valid — that belongs in a status condition, not an endless retry).

Drift correction. Put it all together and the loop is self-healing by construction. Because reconcile recomputes the full diff every time instead of reacting to exactly what changed, it doesn't matter whether drift came from kubectl edit, another controller, or the underlying system changing on its own. The next reconcile — watch-triggered or resync-triggered — sees the same gap and closes it the same way.

Part 3: Building BucketOperator

With the concepts in place, let's build something real: BucketOperator, which manages a StorageBucket custom resource backed by a mock object-storage provider — a small HTTP service we'll write too, standing in for a real cloud storage API.

apiVersion: storage.avalonhosting.example/v1
kind: StorageBucket
metadata:
  name: media-assets
spec:
  region: us-east-1
  sizeGB: 100
  encrypted: true
status:
  phase: Active
  id: bucket-482913

The goal: kubectl apply a YAML file and get a real storage bucket, without touching a separate cloud console or CLI. Once it exists, it's just another object the cluster's own tooling — kubectl, RBAC, GitOps pipelines — already knows how to work with.

Setup

kind is the fastest way to get a disposable local cluster to build against:

kind create cluster --name bucketoperator

We'll write the operator itself in Go, point kubectl at the new cluster, and write the mock provider in Python with Flask:

pip install flask

The Mock Storage Provider

Before writing any controller code, we need something for it to control. The mock provider is a small HTTP service with three endpoints — POST /buckets to create one, GET /buckets/{id} to check on it, and DELETE /buckets/{id} to remove it — backed by nothing more than a dictionary in memory. Every bucket it creates starts in Provisioning and flips to Active a few seconds later on its own, which forces our reconciler to actually poll rather than assume success:

import random
import string
import threading
import time
from flask import Flask, jsonify, request

app = Flask(__name__)
buckets = {}  # in-memory store, keyed by bucket id

def provision(bucket):
    time.sleep(5)  # simulate provisioning taking time
    bucket["phase"] = "Active"

@app.post("/buckets")
def create_bucket():
    body = request.get_json()
    bucket_id = "bucket-" + "".join(random.choices(string.digits, k=6))
    bucket = {"id": bucket_id, "region": body["region"], "phase": "Provisioning"}
    buckets[bucket_id] = bucket
    threading.Thread(target=provision, args=(bucket,), daemon=True).start()
    return jsonify(bucket)

@app.get("/buckets/<bucket_id>")
def get_bucket(bucket_id):
    bucket = buckets.get(bucket_id)
    if bucket is None:
        return "", 404
    return jsonify(bucket)

@app.delete("/buckets/<bucket_id>")
def delete_bucket(bucket_id):
    buckets.pop(bucket_id, None)
    return "", 204

if __name__ == "__main__":
    app.run(port=8090, threaded=True)

Run this as its own process alongside the cluster, on the port the operator will call. It knows nothing about Kubernetes at all — which is the point. It stands in for wherever a real provider API would sit.

Defining the StorageBucket CRD

The Go type follows the spec/status split from Part 2 directly:

type StorageBucketSpec struct {
    Region    string `json:"region"`
    SizeGB    int    `json:"sizeGB"`
    Encrypted bool   `json:"encrypted"`
}

type StorageBucketStatus struct {
    ID    string `json:"id,omitempty"`    // provider-assigned id, empty until first provisioned
    Phase string `json:"phase,omitempty"` // mirrors the provider's lifecycle phase
}

type StorageBucket struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
    Spec   StorageBucketSpec   `json:"spec,omitempty"`
    Status StorageBucketStatus `json:"status,omitempty"`
}

type StorageBucketList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []StorageBucket `json:"items"`
}

TypeMeta carries kind and apiVersion — the same two fields every Kubernetes object has, built-in or custom. ListMeta is its counterpart on list types, carrying resourceVersion and pagination fields instead.

Every registered type needs to satisfy runtime.Object, meaning it needs a DeepCopyObject method. This is normally code-generated, but here's what that generated code looks like by hand — the rest follow the same mechanical shape:

func (in *StorageBucket) DeepCopyObject() runtime.Object {
    out := StorageBucket{
        TypeMeta:   in.TypeMeta,
        ObjectMeta: *in.ObjectMeta.DeepCopy(), // ObjectMeta already knows how to copy itself
        Spec:       in.Spec,                   // no pointers or slices in Spec, a plain copy is safe
        Status:     in.Status,
    }
    return &out
}

And the manifest that teaches the API server about the type:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: storagebuckets.storage.avalonhosting.example
spec:
  group: storage.avalonhosting.example
  scope: Namespaced
  names:
    kind: StorageBucket
    listKind: StorageBucketList
    plural: storagebuckets
    singular: storagebucket
    shortNames: [sbucket] # lets us type `kubectl get sbucket`
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {} # splits status into its own subresource, per Part 2
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [region, sizeGB]
              properties:
                region: { type: string }
                sizeGB: { type: integer }
                encrypted: { type: boolean }
            status:
              type: object
              properties:
                phase: { type: string }
                id: { type: string }

The subresources.status line is what actually enforces the spec/status boundary from Part 2 — status becomes a separate write path with its own permissions. The names block is what kubectl resolves against: plural is why kubectl get storagebuckets works, and shortNames is why kubectl get sbucket works too.

Writing the Reconciler

The reconciler's job is small to describe: look at a StorageBucket, make sure a matching bucket exists at the provider, and keep status honest. We wrap the provider's HTTP API behind a small client so the reconciler itself stays readable:

func (r *StorageBucketReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var bucket storagev1.StorageBucket
    if err := r.Get(ctx, req.NamespacedName, &bucket); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err) // object was deleted, nothing left to do
    }

    if bucket.Status.ID == "" {
        // no bucket yet, this is the first time we've seen this object
        created, err := r.Provider.Create(ctx, bucket.Spec.Region)
        if err != nil {
            return ctrl.Result{}, err
        }
        bucket.Status.ID = created.ID
        bucket.Status.Phase = created.Phase
        if err := r.Status().Update(ctx, &bucket); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{RequeueAfter: 3 * time.Second}, nil // check back shortly instead of blocking here
    }

    // bucket already exists, poll the provider for whatever it knows right now
    current, err := r.Provider.Get(ctx, bucket.Status.ID)
    if err != nil {
        return ctrl.Result{}, err
    }
    bucket.Status.Phase = current.Phase
    if err := r.Status().Update(ctx, &bucket); err != nil {
        return ctrl.Result{}, err
    }
    if current.Phase != "Active" {
        return ctrl.Result{RequeueAfter: 3 * time.Second}, nil // still provisioning, keep polling
    }
    return ctrl.Result{}, nil
}

Two things worth calling out: this only fires at all because we've registered a watch on StorageBucket — the API server tells us the moment one is created or edited. And every branch ends by writing to bucket.Status. Kubernetes never talks to the provider directly; the only way anyone learns a bucket is active is because our reconciler wrote that into status.

Failure Handling and Retries

Notice the reconciler above never retries anything itself. When r.Provider.Create or r.Provider.Get fails — a network blip, the mock provider not being up yet — it just returns the error. That's deliberate: returning an error is how we ask controller-runtime to requeue with exponential backoff on our behalf, so a persistently unreachable provider doesn't get flooded with retries.

The one thing to be careful of is treating every failure identically. A timeout talking to the provider is worth retrying; a StorageBucket whose spec.region the provider will never accept is not — retrying that forever just produces a busy loop that never succeeds. (Surfacing that distinction through a status condition is a natural next exercise once the core loop above feels solid; our mock provider never rejects a request outright, so it doesn't come up here.)

Adding a Finalizer

Delete a StorageBucket right now, and Kubernetes removes the object while leaving an orphaned bucket behind at the provider. A finalizer closes that gap — it's a string on the object that tells Kubernetes "don't actually delete this until I say so":

const bucketFinalizer = "storage.avalonhosting.example/bucket-cleanup"

func (r *StorageBucketReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var bucket storagev1.StorageBucket
    if err := r.Get(ctx, req.NamespacedName, &bucket); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    if !bucket.DeletionTimestamp.IsZero() {
        // being deleted, deprovision through the provider before letting it go
        if controllerutil.ContainsFinalizer(&bucket, bucketFinalizer) {
            if bucket.Status.ID != "" {
                if err := r.Provider.Delete(ctx, bucket.Status.ID); err != nil {
                    return ctrl.Result{}, err
                }
            }
            controllerutil.RemoveFinalizer(&bucket, bucketFinalizer) // safe to let the delete proceed now
            return ctrl.Result{}, r.Update(ctx, &bucket)
        }
        return ctrl.Result{}, nil
    }

    if !controllerutil.ContainsFinalizer(&bucket, bucketFinalizer) {
        controllerutil.AddFinalizer(&bucket, bucketFinalizer) // register before we ever provision anything
        if err := r.Update(ctx, &bucket); err != nil {
            return ctrl.Result{}, err
        }
    }
    // ... provisioning logic from before
    return ctrl.Result{}, nil
}

A kubectl delete against a StorageBucket carrying this finalizer doesn't remove it — it sets deletionTimestamp and waits. Our reconciler sees that on the next call, deprovisions the bucket through the provider, and only then removes the finalizer, at which point Kubernetes finally deletes the object. Skip the finalizer and there's no guarantee cleanup ever runs.

Avoiding Reconcile Loops with a Predicate

There's a subtle bug already sitting in the reconciler above: every call to r.Status().Update is itself a change to the object, which triggers our own watch, which calls reconcile again. Left alone this doesn't spin forever — it settles once status stops changing — but it's still wasted work reconciling in response to our own writes.

A predicate filters which events actually enqueue a reconcile, before our code ever runs:

func (r *StorageBucketReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&storagev1.StorageBucket{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})). // drop status-only events
        Complete(r)
}

generation only increments when spec changes — status updates don't touch it. GenerationChangedPredicate uses that to drop events where nothing but status moved, so our own writes stop retriggering us, and we're back to reconciling only on meaningful changes or explicit requeues.

Owned Resources

A bucket sitting somewhere external isn't very useful on its own, so let's have the operator also create a Secret holding its connection details in-cluster:

func (r *StorageBucketReconciler) reconcileAccessSecret(ctx context.Context, bucket *storagev1.StorageBucket) error {
    secret := &corev1.Secret{
        ObjectMeta: metav1.ObjectMeta{
            Name:      bucket.Name + "-access",
            Namespace: bucket.Namespace,
        },
        StringData: map[string]string{"id": bucket.Status.ID},
    }
    if err := controllerutil.SetControllerReference(bucket, secret, r.Scheme); err != nil {
        return err // ties the Secret's lifecycle to this StorageBucket
    }
    return r.Patch(ctx, secret, client.Apply, client.ForceOwnership, client.FieldOwner("bucketoperator")) // create or update, either way
}

SetControllerReference is what makes this an owned resource — it stamps an owner reference onto the Secret pointing back at the StorageBucket. Two things fall out of that for free: deleting the StorageBucket now cascades (Kubernetes garbage-collects the Secret automatically), and no finalizer is needed for it, since it's an in-cluster object rather than an external one.

Add Owns(&corev1.Secret{}) alongside For(&storagev1.StorageBucket{}) in SetupWithManager, and an edit or deletion of the Secret itself re-triggers reconciliation of its owning StorageBucket — so if someone deletes it by hand, we notice and recreate it. The same pattern, called a second time for a Service fronting the bucket in-cluster, is all it takes to own a second resource type. There's nothing more to owning multiple resources than repeating this once per type.

Cross-Resource Reconciliation

Every StorageBucket so far talks to one hardcoded provider endpoint. Real deployments need that configurable, and it's rarely a one-off — fifty buckets in one account share the same endpoint and credentials, and a hundred more might live under a different account entirely.

We could put an endpoint field directly on StorageBucketSpec, but then rotating a credential means editing every bucket that uses it, one at a time. Pulling that into its own object lets many buckets reference it by name instead, so a single edit propagates everywhere:

type BucketProviderConfigSpec struct {
    Endpoint string `json:"endpoint"`
}

Add a providerRef field on StorageBucketSpec pointing at one by name. The interesting part isn't the new type — it's what happens when a BucketProviderConfig changes. A StorageBucket doesn't watch it directly, and there's no owner reference between them, so a plain Owns() won't catch it. Instead, we watch the type and map each event onto every StorageBucket that references it:

func (r *StorageBucketReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&storagev1.StorageBucket{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
        Owns(&corev1.Secret{}).
        Owns(&corev1.Service{}).
        Watches(
            &storagev1.BucketProviderConfig{}, // not owned, so Owns() won't catch its changes
            handler.EnqueueRequestsFromMapFunc(r.findBucketsForProviderConfig),
        ).
        Complete(r)
}

func (r *StorageBucketReconciler) findBucketsForProviderConfig(ctx context.Context, obj client.Object) []reconcile.Request {
    var buckets storagev1.StorageBucketList
    if err := r.List(ctx, &buckets, client.InNamespace(obj.GetNamespace())); err != nil {
        return nil
    }
    var requests []reconcile.Request
    for _, bucket := range buckets.Items {
        if bucket.Spec.ProviderRef == obj.GetName() { // only re-enqueue buckets that actually reference this config
            requests = append(requests, reconcile.Request{NamespacedName: client.ObjectKeyFromObject(&bucket)})
        }
    }
    return requests
}

This is cross-resource reconciliation: one resource's change causing a different resource type entirely to reconcile, connected only by a field value rather than ownership. In a production version of this operator, BucketProviderConfig is exactly where real credentials and a real endpoint for something like S3-compatible storage would live — the mock provider stands in for precisely that boundary.

RBAC

None of the above works without permission to act on it. The manifest just needs to list what we actually touch — StorageBucket and BucketProviderConfig objects, the status subresource separately, and the Secret/Service objects we create:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bucketoperator-manager-role
rules:
  - apiGroups: ['storage.avalonhosting.example']
    resources: ['storagebuckets', 'bucketproviderconfigs']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
  - apiGroups: ['storage.avalonhosting.example']
    resources: ['storagebuckets/status'] # separate rule, it's a separate subresource
    verbs: ['get', 'update', 'patch']
  - apiGroups: ['']
    resources: ['secrets', 'services']
    verbs: ['get', 'list', 'watch', 'create', 'update', 'patch', 'delete']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bucketoperator-manager-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: bucketoperator-manager-role
subjects:
  - kind: ServiceAccount
    name: bucketoperator-controller-manager
    namespace: bucketoperator-system

Note: BucketOperator manages a resource entirely outside the cluster through a hand-rolled HTTP client, but that's not an unusual pattern — Crossplane, AWS Controllers for Kubernetes, Cluster API, and cert-manager all reconcile external or non-Kubernetes state through CRDs the same way, and are worth reading once this pattern feels familiar. We're also keeping BucketOperator's scope deliberately narrow here — resizing a bucket, supporting more than one real provider behind BucketProviderConfig, and similar features are natural extensions once the core loop feels solid.

Part 4: Getting to Production

Packaging and Deployment

Everything so far has run as a binary on your own machine — go run against whatever cluster kubectl happens to be pointed at. A real Deployment needs an image instead, so the operator gets a multi-stage Dockerfile: one stage to compile, and a much smaller image to actually run it:

FROM golang:1.26 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /bucketoperator ./cmd/manager

FROM gcr.io/distroless/static-debian12
COPY --from=build /bucketoperator /bucketoperator
USER 65532:65532 # nonroot, matches the security context on the Deployment below
ENTRYPOINT ["/bucketoperator"]

The build stage carries the full Go toolchain and every source file, none of which need to ship. The final image only has the compiled binary — there isn't even a shell to get a foothold in, before runAsNonRoot is set at all:

docker build -t registry.example.com/bucketoperator:v0.1.0 .
docker push registry.example.com/bucketoperator:v0.1.0

With that image pushed somewhere the cluster can pull from, the rest of the manifests go on — CRDs first, since the operator's Deployment will crash-loop if it starts and immediately tries to watch a resource type the API server has never heard of:

kubectl apply -f config/crd/
kubectl apply -f config/rbac/
kubectl apply -f config/manager/

Schema changes are the part hand-written manifests make you feel directly. Adding a field to StorageBucketSpec is harmless — existing objects just don't have it set yet. Renaming or restructuring one isn't, since every stored object was serialized against the old shape. The CRD's versions list exists for exactly this: more than one version can be served at once, one is marked storage to say which shape is actually persisted, and a conversion webhook translates between versions when a client asks for one that isn't the stored one. BucketOperator doesn't need this today since v1 is the only version that's ever existed — but it's why versions was written as a list from the first manifest onward, not a single value.

None of this replaces someone running kubectl apply by hand forever — a CI pipeline that builds the image, pushes it, and applies the manifests on merge to your main branch is the natural next step, and it's ordinary CI/CD once the manifests themselves live in Git.

Performance and Resilience

By default, a controller processes one reconcile at a time. That's fine while you're the only one testing it, but with hundreds of StorageBucket objects, most of them just sit in the workqueue waiting their turn even though reconciling one doesn't block reconciling another. MaxConcurrentReconciles raises that ceiling:

func (r *StorageBucketReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&storagev1.StorageBucket{}, builder.WithPredicates(predicate.GenerationChangedPredicate{})).
        Owns(&corev1.Secret{}).
        Owns(&corev1.Service{}).
        Watches(&storagev1.BucketProviderConfig{}, handler.EnqueueRequestsFromMapFunc(r.findBucketsForProviderConfig)).
        WithOptions(controller.Options{MaxConcurrentReconciles: 8}). // eight buckets in flight instead of one
        Complete(r)
}

Caching only helps one side of this reconciler. Reading bucket back via r.Get is already fast and local — informers keep that in memory. But r.Provider.Get is a real HTTP round trip every time, with nothing caching it. That asymmetry is the same one from Part 2: in-cluster reads are cheap because Kubernetes built the caching layer for us, and external reads cost exactly whatever's on the other end of the wire. A short-lived cache in front of the provider client is tempting, but it comes with a real cost — a cached Active for a bucket that just failed is a lie your status will keep repeating until the cache expires.

Nothing stopping a burst of reconciles from hammering an uncached provider, either — say every StorageBucket getting touched at once after a cluster restart. Wrapping the client in a rate limiter caps that independently of whatever backoff the workqueue already applies on failures:

type Client struct {
    baseURL string
    http    *http.Client
    limiter *rate.Limiter // shared across every reconcile using this client
}

func (c *Client) Create(ctx context.Context, region string) (*Bucket, error) {
    if err := c.limiter.Wait(ctx); err != nil {
        return nil, err
    }
    // ... existing HTTP call
}

Leader election is the other half of running more than one replica safely — two fields on the manager:

mgr, err := ctrl.NewManager(cfg, ctrl.Options{
    LeaderElection:   true,
    LeaderElectionID: "bucketoperator-leader",
})

With this set, every replica starts, but only the one holding the lease actually reconciles; the rest sit ready to take over the moment it stops renewing in time.

Concurrency also surfaces a race worth naming directly. Say the reconciler calls r.Provider.Create, the provider creates the bucket and returns its id, and then the process crashes before r.Status().Update ever runs. bucket.Status.ID is still empty, so the next reconcile sees an object with no bucket yet and calls Create again — now the provider has two buckets for one StorageBucket. Neither MaxConcurrentReconciles nor leader election prevents this; it's a gap in the create step itself, and it only shows up once something can fail between the external call and the write that records it. Closing it for real means the provider needs to accept an idempotency key, generated once and stored on the object before the first Create call, so a retried create recognizes it already happened instead of provisioning a second bucket.

Security

The ClusterRole from Part 3 works, but it's broader than it needs to be — it grants every verb on Secrets and Services cluster-wide, when the operator only ever touches the ones it owns. A tighter version scopes to a single namespace with Role/RoleBinding instead of ClusterRole/ClusterRoleBinding wherever BucketOperator only ever runs in one, and drops verbs it never actually calls — it never lists or watches arbitrary Secrets outside the ones it creates.

Credentials are the other gap. BucketProviderConfig currently holds a plaintext endpoint, and a real provider needs an API key alongside it — which has no business sitting in a CRD spec anyone with read access can see. It belongs in a Secret, referenced by name instead of embedded:

type BucketProviderConfigSpec struct {
    Endpoint  string                      `json:"endpoint"`
    SecretRef corev1.LocalObjectReference `json:"secretRef"` // Secret holding the provider's API key
}

The reconciler resolves SecretRef at the point it builds the provider client, reads the key out of the Secret's data, and never logs it or writes it back to anything with wider read access — including the StorageBucket's own status.

The last piece is the operator's own pod. A container security context that runs as a non-root user, sets a read-only root filesystem, and drops Linux capabilities it doesn't need shrinks what's possible if the binary itself is ever compromised — standard practice for any workload, not something operator-specific. (If you add an admission webhook anywhere in your own version of this, its certificates would belong in this same security review.)

Observability

The manager exposes a Prometheus endpoint without any extra work — workqueue depth, reconcile duration, and reconcile error counts are already there per controller. What isn't automatic is anything about the provider itself, so add a metric the way any Go service would:

var providerCallDuration = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "bucketoperator_provider_call_duration_seconds",
        Help: "Duration of calls to the storage provider, by operation",
    },
    []string{"operation"},
)

func init() {
    metrics.Registry.MustRegister(providerCallDuration) // shares the manager's existing /metrics endpoint
}

Wrapping each provider call with a timer around this turns "is the provider slow?" from a guess into something you can graph. Logging benefits from the same instinct: log.FromContext(ctx) inside Reconcile already carries the object's name and namespace on every line once that's set up in SetupWithManager. Adding bucket.Status.ID to that logger right after it's set means every later log line for that reconcile also carries the provider's own identifier — which is what makes it possible to grep a mock provider log and an operator log for the same request and find both sides of the same failure.

Common Issues

The operator's Deployment crash-loops on startup. This almost always means the CRD wasn't applied first. Apply config/crd/ before config/manager/, always.

Reconcile keeps firing even though nothing meaningful changed. Check whether a GenerationChangedPredicate is actually wired into SetupWithManager — without it, your own status writes will keep retriggering reconciliation.

Deleting a custom resource hangs forever. A finalizer is present but never getting removed — usually because the deprovisioning call inside the finalizer branch is failing silently, or the reconciler isn't handling the DeletionTimestamp branch at all.

Two external resources exist for what should be one custom resource. This is the create-then-crash race described above — the fix is an idempotency key passed to the provider on create, not a retry-loop workaround.

Related Articles / Next Steps

• [Link to an Avalon guide on deploying containerized applications, if available]
• [Link to an Avalon guide on Kubernetes hosting or managed cluster options, if applicable]
• [Link to Avalon's container/VPS hosting product page, if applicable]