Backups in a GitOps cluster fleet
We restored a Deployment from last night's backup and watched the cluster delete it in real time. No warning. No conflict dialog. The controller saw drift and fixed it, and the restore we'd run to save ourselves just made the incident worse. That's the moment you learn that backups in a GitOps fleet are not the backups you're used to, and the rule that keeps you out of that room is simpler than the tooling makes it sound.
The short answer. Back up only what Git can't rebuild. Deployments, Services, ConfigMaps and SOPS-encrypted Secrets are recreated by the controller from Git, so backing them up is waste and actively harmful, because the restore is a drift event. Velero is for the bytes Git can't reconstruct, and the management cluster is the one exception that gets backed up completely. Read "Flux" as "your GitOps controller" throughout, and the argument holds for Argo CD too, because the reconciler is the part doing the work and both behave the same way on drift.
Git is the backup, for everything Git manages
This is the shift that takes a while to internalize. In a GitOps fleet, the repository is the backup for everything declarative, with one word of precision. Git is the backup for everything Git manages. Deployments, Services, ConfigMaps, the SOPS encrypted Secrets, all of it gets recreated by the controller from Git. You don't need to back up what you can rebuild from the source of truth, and backing it up actively works against you, because the restore is a drift event, not a recovery.
The precision matters, because the things Git does not manage are exactly the things a backup must cover. Certificate private keys that cert-manager generated, Custom Resources an operator created at runtime, ServiceAccount token data, controller generated Secrets. None of those are in the repo. A PV only backup silently loses all of them. So the honest version of the rule is not "back up only bytes." It's "back up only what Git cannot rebuild, and know the list of what that actually is on your fleet."
Same logic for the secret tooling. SealedSecrets, External Secrets Operator, Vault agent, whatever you use, the pattern is identical. The encrypted secret lives in Git and is rebuildable, the decryption key does not and is not. Back up the key, not the encrypted blob. The blob is Git's job.
One category to check before you trust the model. CRDs. If you manage them through the controller, they're in Git and they rebuild. If you installed them manually, or a Helm chart manages them outside the GitOps flow, they're not in the repo, and a fresh cluster won't have them. Your CRD strategy decides whether the declarations that use them can come back. Worth an audit pass before you need it. Same for Namespace objects, with the same caveat. They're as declarative as anything else, but a lot of fleets create them by hand, through Helm, or through Cluster API itself, which means they never touched Git. Verify your namespace strategy too, because a namespace that exists only in a live cluster is one restore away from not existing at all.
So what actually needs backing up
The question is not "what do we back up." It's "what does Git not own." And the answer is bytes. Persistent volume data, the database, the stateful workload's disk. That is the only thing in the workload cluster that exists outside the Git model, and it is the only thing Velero should be protecting there.
The restore is a split. You restore the PV bytes from the snapshot, and the controller supplies the declaration from the current Git state. No conflict, because you never restore the Deployment in the first place. Data from the snapshot, declaration from Git, and the cluster ends up exactly where it should be.
One honest caveat on the data side, because "restore the PV bytes" sounds simpler than it is. A plain filesystem snapshot is not crash consistent for a database. Postgres and MySQL want to be quiesced or they'll replay on restore, and Velero's pre and post hooks exist for exactly this, running a pg_dump or a flush before the snapshot and a cleanup after. The split is right. The data half is where the actual work lives.
And one level up, the repo itself needs a seat at the DR table. If the cluster dies, Git rebuilds it. If Git dies, nothing rebuilds anything. The repository is a control plane now, and it needs mirrors and off site copies the way any other control plane does. A lost GitHub account is a fleet incident, not a code incident.
The exception, the management cluster
There is one cluster where none of this applies, and it's the one people instinctively skip. The management cluster, the one running the GitOps controllers and the Cluster API machinery, is not GitOps managed itself. It's the thing doing the managing, so Git has nothing to rebuild it from. If Cluster API is new to you, the official glossary defines it plainly. The management cluster is the Kubernetes cluster where Cluster API runs its controllers, and workload clusters are what those controllers create and reconcile.
Its etcd holds state nothing else has. The controllers' leases, the Cluster API objects, the SOPS key secrets, the Git credentials. That cluster gets traditional backups, etcd snapshots included, full stop. This is the one place the old drill is still right, because it's the one cluster the reconciler can't rebuild.
What deliberately not to back up
- Anything the controller manages. It's in Git. Restoring it creates drift and a reconcile fight you will lose.
- flux-system itself. Re bootstrap it with
flux bootstrapfrom the repo. It's designed to be rebuilt, with one catch. The Git credentials and the SOPS or age key secrets live in the cluster, not in the repo. Restore those first, or the fresh bootstrap can't pull a private repo or decrypt anything. - etcd snapshots of managed state. Restoring a snapshot into a reconciling system invalidates tokens, resurrects stale status, and reverts anything Git disagrees with. Keep it only for the cluster so broken Git can't rebuild it, and know you're accepting a fight.
That last one is where the etcd dogma dies, and it's worth being fair about why. A snapshot of a workload cluster's etcd is the only thing that recovers the full API state in one shot, Secrets, leases, ServiceAccount tokens, Custom Resource status. That is also exactly why restoring it into a managed cluster is a trap. The token controller invalidates restored ServiceAccount tokens. The controllers see restored status that disagrees with the objects they own and overwrite it. And the reconciler reverts anything Git disagrees with. A full state restore of a managed cluster does not recover the cluster, it generates a second incident and then the drift war on top.
So the split lands in an unexpected place. Workload clusters under GitOps, back up only the bytes, because the reconciler is the recovery mechanism and a snapshot fights it. The management cluster, back up everything, because it's the one cluster Git can't rebuild. Same fleet, opposite answers, and the boundary between them is exactly the line between what the controller owns and what it runs on.
When the model breaks
This whole split only holds if your team actually keeps declarations out of the cluster and in Git. Real fleets have drift in practice. Manual kubectl, helm rendered state that never got committed, operators writing their own objects. If that's your fleet, "Git owns everything" is the dream, not the audit, and your backup scope needs to cover the gap until the hygiene catches up.
And the restore is not a parallel operation. The management cluster has to come back first, because workload clusters reconcile from it, and a workload cluster has nothing to reconcile against until it exists. Beyond that, fleet dependencies impose their own order. Shared configuration, hub and spoke networking, clusters that consume another cluster's services, all of it means you restore in dependency order, not alphabetical order. The good news is that the order falls out of the same Git structure, because the overlays reference each other. Restore the foundation, then the layers.
The boring conclusion
Back up the data, not the declaration. Git is the declaration's backup. The shorter your list of "things Git doesn't own" is, the simpler your DR story is, and the less you pay for storage you'll never safely restore.
Have you ever restored a manifest and watched your GitOps controller undo it in real time?
Frequently asked questions
How do you back up a GitOps cluster?
Back up only what Git cannot rebuild. Deployments, Services, ConfigMaps and SOPS-encrypted Secrets are recreated by the controller from Git, so backing them up with Velero is waste and actively harmful, because the restore is a drift event. Velero is for the bytes Git can't reconstruct, the persistent volume data, keys, CRs and CRDs.
What does Velero back up in a GitOps fleet?
Only the data Git doesn't own. Persistent volume data, databases, stateful workload disks, certificate private keys, runtime-created Custom Resources, ServiceAccount token data and controller-generated Secrets. The declaration comes from Git, the bytes come from the snapshot.
Is the management cluster backed up differently?
Yes. The management cluster is the one cluster Git can't rebuild, because it runs the GitOps controllers and Cluster API machinery. It gets traditional backups, etcd snapshots included, full stop. Its etcd holds the controllers' leases, Cluster API objects, SOPS keys and Git credentials.