Kafka backup best practices for production environments

TL;DR: Production-grade Kafka backup means an independent, immutable copy of your topics, consumer offsets and schemas, stored outside the cluster, with point-in-time restore and RPO/RTO targets the business has signed off on. Replication is not backup. If you have never run a restore, you do not have a backup, you have a hope.

What makes a Kafka backup "production-grade"?

Most teams get to a working backup fairly quickly. A Connect sink writing to S3, a nightly script, a second cluster fed by MirrorMaker 2. All of that copies bytes somewhere else, and on a good day it looks like it actually works.

Reality shows up on a bad day. A production-grade setup is one that survives the failure modes Apache Kafka itself does not protect you against, and that can be executed under pressure by someone who was not the person who built it.

Concretely, four things separate the two.

It captures more than topic data. A restore that brings back the events but not the consumer group offsets forces every consumer to reprocess or skip, which is its own incident. The same goes for schemas: restore into an environment with a different Schema Registry and the schema IDs no longer line up, so messages fail to deserialise. Topic data, consumer offsets and schemas belong in the same backup scope.

It is decoupled from the cluster. Backups that live on the same storage layer, in the same region, under the same credentials as the brokers share a blast radius with the thing they are supposed to protect. Tiered storage (KIP-405) is a good example of something that looks like a safety net and is not one. Offloaded segments are still owned by the live cluster, so a topic delete or a retention change cascades straight through to the remote tier.

It supports point-in-time restore. "Restore the last snapshot" only helps for total loss. The more common incidents are narrower: a producer pushed corrupt data for forty minutes, someone lowered retention.ms on the wrong topic, a bad deploy poisoned a partition. Recovering from those means rolling a topic back to a state before the damage, not replaying everything since the beginning of time.

It is documented and rehearsed. A restore procedure that only exists in one engineer's head is a single point of failure with a mortgage and a holiday allowance.

If you are still working out the fundamentals of why Kafka needs a backup at all, start with our Kafka backup guide and come back here.

Why replication keeps getting mistaken for backup

This is the single most common misunderstanding in the Kafka ecosystem. A replication factor of three protects you against losing a broker. It does nothing when a topic gets deleted, when retention is misconfigured, or when a bug writes garbage that gets faithfully copied to all three replicas within milliseconds.

The same logic applies to MirrorMaker 2, Cluster Linking and MSK Replicator. They are availability tools. They mirror your mistakes as efficiently as they mirror your data. You want both layers, and you want to stop pretending one of them is the other.

How do you set the right RPO and RTO for Kafka?

RPO (how much data you can afford to lose) and RTO (how long you can afford to be down) look like engineering numbers. They are not. They are business decisions dressed in technical clothing, and the most common failure is a platform team quietly inventing them because nobody upstream would commit.

Three practices make the difference.

Set them per workload, not per cluster. A single RPO for the whole cluster is almost always wrong. The payment event stream and the clickstream telemetry topic do not have the same value, do not have the same audience, and should not have the same recovery target. Tier your topics. Most clusters have a handful that genuinely matter and a long tail that does not.

Make the business own the number. The question behind RPO and RTO is not technical. Is three days of downtime acceptable? A platform engineer cannot answer that, and neither can a consultant. The business has to. And since the default answer to "how long can you wait" is always "no time at all", the only way to get a real number is to put a price on each option and let the people who own the P&L choose.

Validate the number against reality. An RTO of thirty minutes on a multi-terabyte cluster is meaningless until you have measured how long a restore actually takes with your data volume, your network and your storage tier. Financial institutions running script-based backups routinely discover their real restore time is eight hours or more, which is how a compliance target quietly turns into a compliance gap.

One more thing that people underestimate: restoring Kafka is the easy part. Deciding which applications start first, which ones cannot run in parallel, and who tells the call centre to brace for impact, that is the hard part. It belongs in the playbook, and the playbook belongs to the application teams, not only to the platform team.

Where should Kafka backups actually live?

There is no universally right answer here, only trade-offs you should be making deliberately.

Separate failure domain, always. Same cluster is not a backup. Same account, same region and same credentials as the brokers is a weak backup. Ransomware and mis-scoped IAM policies do not respect the boundary between "production data" and "backup of production data".

Immutability over convenience. Object lock, versioning, write-once retention. If an attacker or a misfiring cleanup job can delete the backup with the same credentials that can delete the topic, you have one copy, not two.

Tier by age. The pattern is familiar from any other backup domain: keep recent backups on warm storage where a restore is fast, move older data to cold storage where retrieval is slower and cheaper. On AWS that is S3 into Glacier, on Azure it is Hot into Cool or Archive, on-premises it is whatever your storage team already runs. Watch the minimum-duration billing on archive tiers before you set the lifecycle policy, because deleting cold data early still bills you for the full period.

Ask the jurisdiction question early. Where the bytes physically sit, who can subpoena them, and which cloud provider's control plane can read them are questions that surface during a compliance review whether or not you planned for them. Cheaper to answer them at design time.

Watch out for bucket-level retention policies. If your backup tool keeps its own index of what exists in storage, a lifecycle rule silently deleting objects underneath it will corrupt that bookkeeping. Retention should be configured in the backup tool, not bolted on at the bucket.

There is more detail on weighing these options in choosing the right backup strategy for Kafka-based environments.

How Kannika applies these practices

Kannika Armory runs as a Kubernetes operator inside your own cluster. Backups are defined as Custom Resources, so they fit into GitOps pipelines and version control instead of living in a cron job somewhere. It continuously captures topic data, consumer group offsets and Schema Registry schemas, and writes them to storage you control: S3 and S3-compatible endpoints such as MinIO or Ceph, Azure Blob Storage, Google Cloud Storage, or Kubernetes PersistentVolumes for fully on-premises setups.

Restores are point-in-time and selective. You can bring back a single topic as it existed five minutes ago, filter on timestamp, rename topics on the way in, and let Armory remap schema IDs and translate consumer offsets so applications resume where they left off instead of reprocessing. Data is compressed on the way to storage, typically to a fraction of its original volume, which is what makes keeping longer history affordable. It works against Apache Kafka, Confluent, MSK, Azure Event Hubs and Redpanda.

How often should you test your Kafka backup?

At minimum once a year, in a scheduled drill, with the people who would actually run it. If Kafka carries data that falls under DORA or NIS2, that annual cadence is not a suggestion. DORA Article 11 requires financial entities to test continuity and recovery plans at least yearly, and for anything above microenterprise scale those tests have to include cyber-attack and switchover scenarios. Article 12 requires that restoration from backups is tested periodically. NIS2 Article 21 puts backup management and disaster recovery squarely inside the required risk-management measures. We unpack what that means for Kafka specifically in NIS2, DORA and Apache Kafka.

Compliance is the floor. The engineering argument is stronger:

"Nobody cares about backups. Everyone cares about recovery. If you back up every day and never recover, how do you actually know your recovery works?"

, Anatoly Zelenin

A useful test is not a green checkmark in a dashboard. It is a restore into a clean environment, with a stopwatch, measured against your stated RTO, run by someone other than the person who built the setup. Environment cloning helps here, because the restore you use to spin up a staging environment from production data is the same operation you would run in an incident. Teams that clone environments regularly get their recovery testing almost for free.

And the timing matters as much as the test itself. There are two moments when you can discover your recovery does not work: a Wednesday afternoon with the whole team at their desks, or Christmas Eve. Netflix picked the second one by accident, which is how Chaos Monkey came to exist. Banks running two datacenters have known this for years and schedule an annual cutover drill to see what breaks. Kafka deserves the same treatment.

Production-grade backup compared to the usual alternatives

For the broader why-and-how of Kafka backup, see our Kafka backup guide. For how Kannika Armory stacks up against a Connect sink or a custom script at the mechanics level, see how to backup and restore a Kafka topic. If you're specifically on Confluent, see Confluent and Kafka backup for how Cluster Linking, Replicator, and Kannika Armory stack up in that environment. The full, regularly updated version of this comparison lives on our product page.

What are the most common Kafka backup mistakes?

• Treating replication as backup. Covered above, and still the number one cause of teams discovering they have no recovery path at the worst possible moment.
• Backing up data but not offsets or schemas. The restore technically succeeds and the applications still break.
• One RPO for the whole cluster. Either you overpay to protect telemetry, or you under-protect the topics that actually matter.
• Backups in the same blast radius. Same region, same account, same credentials, mutable storage.
• Never testing. The plan exists, it is in Confluence, it has been reviewed. It has never been run.
• Buying a tool and calling it a strategy. A licence, ours included, does nothing for an organisation that has not mapped its failure scenarios, agreed its targets and written its playbook. Do that work first, then pick the tool.
• No named owner. Backups without an owner drift out of scope as topics get added. New topic, no backup policy, nobody noticed.

Ready to see what this looks like in practice?

Book a 30-minute demo and we will walk through your setup, your RPO and RTO targets and where the gaps are. Or start a free trial and back up your first topic today. No credit card required.

For the wider picture on why Kafka needs backup in the first place, read our Kafka backup guide.