The GitOps fleet gap
The case for GitOps isn't a feature list. It's a 3am scenario. Someone deletes a running Deployment, and in a push world that deletion sticks until a human wakes up and types the workload back into existence. In a GitOps fleet the controller diffs live state against Git, rebuilds the Deployment, and nobody's phone rings. The whole incident class just stops existing. That asymmetry is the whole argument. Because once you see it, the next step was never another command. It was a decision about everything that has to exist before the command could mean anything. That list, the stuff between the one liner and a real fleet, is the gap this piece is about.
The short answer. The gap is everything that has to exist before your first declarative cluster, and none of it is glamorous.
I've stood on both sides of that gap. The greenfield side is a single
eksctl create cluster or a kubeadm init and you're done. Ten minutes, a
working cluster, zero paperwork. It feels like the honest way to run Kubernetes, because
it is, for one cluster that nobody else depends on.
Then the requirements change. You need source control on the whole thing. You need the cluster to be reproducible, reviewable, auditable. You need a second cluster, and a third, and a platform team starts asking what's actually running where. The one liner can't answer any of that, and the gap between it and a real GitOps fleet is wider than most people expect before they try.
This is the list of everything that has to exist before your first declarative fleet. Not the nice to haves. The stuff that genuinely has to be there, in the order it has to be there. Here's the whole stack at a glance, bottom to top.
The management cluster is a single point of failure
The first surprise is that Cluster API needs a cluster to manage clusters. Not a special kind of cluster. Any Kubernetes cluster, running the Cluster API controllers and the provider you picked. For AWS that's CAPA, the Cluster API Provider AWS, and the bootstrap machinery that turns a declarative request into a real EKS cluster. The Cluster API glossary calls this the management cluster, and the clusters it creates are the workload clusters. Worth knowing the official names, because they show up in every object and every error message.
That management cluster is not decorative. It is the thing every workload cluster's reconciliation depends on. If it dies, CAPI level reconciliation stops. Nothing rolls, nothing self heals at the fleet level. To be fair to the nuance, your running workloads keep running, their own controllers keep working, nothing gets reverted. What stops is the management of the fleet, and for a platform team that's the same thing as an outage. The decision between a kind cluster and a real EKS control plane for this role is a decision about the availability of everything else you will ever run.
Kind is a disposable control plane. It is perfect for a lab, for a CI job, for the
official pivot path. clusterctl init on kind, provision real clusters,
then clusterctl move the Cluster API state to a durable cluster and
delete the kind one. The management plane is a staging pod, not a pet.
EKS is a durable control plane. It costs money. It buys you the AWS managed API server and etcd, which means the thing everything depends on does not depend on your laptop. None of this is an AWS pitch. GKE, AKS, or a self managed cluster with real HA fills the same role. The vendor is a preference. The durability at fleet scale is a requirement.
And here's the part that surprises people. Before you can run the management cluster you need the underlay. VPC, subnets, NAT gateways, the network plumbing. Tools like Terraform, Pulumi, or Crossplane earn their place here, because the underlay is infrastructure you will rebuild, not something you babysit. One catch. Crossplane is Kubernetes native and needs a cluster to run in. That seed cluster can be tiny, kind on a laptop or a single node on one instance, but it has to exist before Crossplane builds anything. Skip the seed by standing the first layer up directly and you are back in Terraform or Pulumi territory. Cluster API looks like it starts at the cluster, and it actually starts three layers down, at subnets and NAT gateways that aren't declarative in the cluster model at all. The cluster sits on top of a foundation that has to exist first.
Git is the source of truth, so Git has to be protected
GitOps means the repository is the desired state. That has a security consequence most people don't price in until something leaks. The repo is a production system now. It is the thing that decides what runs, and it needs branch protection, review, signed commits, and the same care you'd give a prod deployment pipeline, because it is one. That includes disaster recovery for the repo itself. It's a control plane now, so it needs mirrors or cross account replication the way any other control plane does. GitHub outages are routine enough to plan for. When Git goes down, your clusters keep running on the last state they synced, but no new change lands anywhere, and a fresh rebuild has nothing to reconcile from. Whether the answer is a mirror, a second vendor, or a self hosted Git server standing by, the ordering does not change. Git has to exist before any declarative work does. It is the foundation under the foundation. A GitHub account going away is a fleet incident, not a code incident.
It also means secrets cannot live in the repo in plaintext, and this is where the gap gets real. You still need secrets. Database credentials, registry credentials, API keys, all of it. The answer is not to stop having secrets. It's to stop committing them.
SOPS encrypts the sensitive fields in the values files before they ever reach Git. The key lives in a KMS or with a GPG key, and the Flux controller assumes an IAM role through a service account annotation to fetch it and decrypt at reconcile time. The secret is in Git, encrypted, and the key that opens it is in the cloud and only the controller can reach it.
There's a second pattern worth knowing, fetch at runtime with the External Secrets Operator or Secrets Store CSI, where nothing secret ever touches Git at all. The trade off is easy to blur, so let's be precise. ESO fetches from the store and then creates a normal Kubernetes Secret, so the plaintext still lands in etcd; what it buys you is no secret in Git and rotation without a commit. Secrets Store CSI is the one that fetches at mount time and never materializes a Secret object at all, at the cost of a runtime dependency on the store. Pick the property you need. Simpler and rotation friendly means ESO, no Secret in etcd at all means CSI.
Backups are for data, not declarations
The GitOps model changes what backup means, and this is the one that trips up people coming from a traditional cluster world. If Git is the source of truth for your Deployments, Services, ConfigMaps, then backing those up with Velero is backing up something Flux can already rebuild from Git. Worse, it's actively harmful. Restore an old Deployment and the controller sees drift and reverts it. Your restore fights your own reconciler, and the reconciler wins.
That's not a weakness, it's the recovery strategy. Flux rebuilding from Git is disaster recovery for declarations. Lose a cluster, bootstrap a new one, and the controller rebuilds the manifests from the repo. What it can't rebuild is bytes. Velero in a GitOps fleet is for the data Git can't reconstruct. Persistent volume data, databases, stateful workload disks. Back up the data, not the declaration. Git is the declaration's backup.
Images, Helm, and the registry as a boundary
The fleet needs container images it trusts, and Helm charts it can actually reconcile. Flux's HelmRelease controller turns charts into the same declarative flow as everything else, and the image registry becomes a security boundary because it is the place trust is decided before anything reaches a node.
Pin by digest, not tag. A tag is a pointer that can move; the digest is the bytes. And if you want the registry to be a real boundary, sign images with cosign, attach an SBOM, and verify at admission time before the kubelet ever pulls.
Policy belongs in the same Git as everything else
Network policies, Kyverno, OPA, they all get the same treatment. Declared in Git, reconciled by Flux. And here's a nuance that matters. Flux only prunes objects it previously applied and that vanished from the repo. A policy someone types into a cluster by hand, that was never in Git, is invisible to Flux and stays forever. That's not a bug. It's the known gap, and it's why drift detection has to be a mechanism, not a hope. Kyverno and Gatekeeper both audit against their declared policies, and a regular audit pass, or an admission time check, is how you catch what GitOps never saw. GitOps does not police what it never managed.
The layering matters. Cilium network policies run in the datapath, per packet, continuously. Kyverno and OPA Gatekeeper run at admission, per request, once. One stops packets, the other stops applies. You need both, and they answer different questions.
The boring conclusion
None of this is glamorous. VPCs, branch protection, a KMS key, a Velero schedule, a Kyverno policy. But this is the actual work of GitOps, and it all has to exist before the first declarative cluster looks effortless.
The one liner isn't bad. It's just the wrong tool once the shape changes. The gap between it and a real fleet is not a missing command, it's a missing foundation. The tooling demos the cluster and skips the Terraform plan underneath it. The foundation is where the fleet actually lives, and that's exactly the part the demo leaves out.
Frequently asked questions
Is the management cluster a single point of failure?
Yes. The management cluster is the thing every workload cluster's reconciliation depends on. If it dies, Cluster API level reconciliation stops. Your running workloads keep running, but the management of the fleet stops, which for a platform team is the same thing as an outage.
What was the thing you had to build before your first GitOps cluster that you never saw coming?