Replicating Kafka across clusters with MirrorMaker 2

Apache Kafka / MirrorMaker 2 / Disaster Recovery / Event-Driven Architecture

Why Kafka Cluster Redundancy Doesn’t Solve All Recovery Issues

Over the past few years, Apache Kafka has become a vital component of enterprise information architecture.

It is used for event streaming between microservices, Change Data Capture, operational analytics, and integration of independent information systems.

For example, a retail holding company has a central ERP, several order systems, logistics services, and a corporate Data Warehouse.

After confirming a business transaction, the ERP creates a corresponding business event. Kafka transmits the message to independent consumers.

The delivery system begins the appropriate processing. The analytical platform updates operational metrics.

Other applications perform defined business actions.

In such an architecture, Kafka becomes an important information intermediary.

If brokers are temporarily unavailable, some services may lose the ability to exchange events in a timely manner. To protect against failures of individual servers, Kafka uses partition replication between brokers.

However, this does not mean automatic protection against the failure of an entire data center. For example, an enterprise has three Kafka brokers, but all of them are physically located in the same data processing center.

If a large-scale infrastructure failure occurs, all three nodes may become unavailable simultaneously.

Even if the enterprise has a backup cluster in another data center, the question arises: how to transfer messages between them, store relevant information contracts, and restore consumers after switching? This requires a separate inter-cluster architecture.

On December 16, 2019, Apache Kafka 2.4.0 was released. One of the main innovations was MirrorMaker 2.0 - an inter-cluster replication mechanism based on Kafka Connect, implementing KIP-382.

For enterprises, this opens up new opportunities for building backup event-driven environments, cluster migration, and geographically distributed integration.

However, it is important to correctly understand the limitations of the technology. MirrorMaker 2 helps replicate data streams between clusters, but does not automatically guarantee zero data loss, full transactional consistency, or flawless failover of all enterprise applications.

Additional procedures are required for productive Disaster Recovery.

Kafka Replication and MirrorMaker 2: Different Levels of Protection

First, it is necessary to distinguish between internal and cross-cluster replication.

Internal Kafka Replication

Ensures the creation of an appropriate number of partition log replicas between brokers of the same cluster. For example, a topic has a replication factor of 3.

With proper configuration, this allows withstanding certain broker failures.

Cross-Cluster Replication

Involves transferring defined messages between independent Kafka clusters. For example, from a Kyiv data center to a backup data center in another region.

MirrorMaker 2 can be used for this purpose.

These mechanisms complement each other.

For example, an enterprise might use a replication factor of 3 in each of its two data centers.

And MirrorMaker 2 will provide the necessary cross-cluster flow.

However, it is important to determine how business consumers will recover.

MirrorMaker 2 Architecture

MirrorMaker 2 uses corresponding Kafka Connect connectors to replicate messages and auxiliary metadata.

At the heart of the architecture are several functional components.

MirrorSourceConnector

Transfers records from the source Kafka cluster to the target.

MirrorCheckpointConnector

Generates information about relevant consumer group offsets to support controlled switching.

MirrorHeartbeatConnector

Creates service messages that help assess the availability of relevant paths between clusters.

Offset Syncs

Used to map offsets between the source and target cluster. These mechanisms are provided by the KIP-382 architecture.

Primary Cluster A

ERP Events · Outbox · Consumer Groups

Backup Cluster B

Remote Topics · Checkpoints

MirrorMaker 2 / Kafka Connect

Source · Checkpoints · Heartbeats

Disaster Recovery Runbook

Fencing · Failover · Offset Translation · Reconciliation

Simplified diagram of one-way replication A → B. MirrorMaker 2 can support more complex topologies, but each requires separate analysis. The advantage of the architecture lies in the ability to use standard Kafka Connect mechanisms.

However, operational complexity remains significant.

For example, the enterprise must correctly configure access to both clusters, topic policies, necessary service resources, and monitoring.

Remote Topics: Why Message Names Change

One of the characteristic features of MirrorMaker 2 is the remote topics naming policy. Under the default scheme, a prefix with the name of the source cluster is used.

For example, in cluster A, there is a topic:

orders.events

After replication to cluster B, it may appear as:

A.orders.events

This allows separating local messages from replicated ones.

Why is such a separation needed?

In complex topologies, two active Kafka clusters may exist.

Each has its own local messages. If both write uncontrollably to the same logical stream, issues with order and duplication may arise.

Prefixes allow separating the origin.

Example

This is a conceptual example of a bidirectional model.

For regular Disaster Recovery, an enterprise can use unidirectional replication.

Importance of Contracts

After failover, consumers may need to use a different topic name. Therefore, the corresponding change must be anticipated in advance.

For example, the consumer configuration can be parameterized. However, it should not be assumed that MirrorMaker 2 automatically changes all application consumer configurations after a disaster.

This is the task of the corresponding recovery procedure.

MirrorMaker 2: Basic Configuration

For Kafka 2.4, a separate configuration file and corresponding MirrorMaker startup can be used.

Here is a simplified example of one-way replication.

# Kafka clusters
clusters = A, B

# Cluster endpoints
A.bootstrap.servers = kafka-a1:9092,kafka-a2:9092
B.bootstrap.servers = kafka-b1:9092,kafka-b2:9092

# A → B replication
A->B.enabled = true
A->B.topics = orders.events,customers.events
A->B.groups = analytics-.*

# No reverse flow
B->A.enabled = false

# Replication and internal topics
replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3
offset-syncs.topic.replication.factor = 3

This is a configuration example that requires adaptation and verification with a specific Kafka 2.4 distribution. It assumes a sufficient number of brokers for the corresponding replication factor and correctly configured security.

For a production infrastructure, it is necessary to additionally define Kafka Connect parameters, authentication, TLS, topic restrictions, and other characteristics.

Starting the corresponding process can be done via:

bin/connect-mirror-maker.sh config/mirror-maker.properties

This startup method is provided by the MirrorMaker 2 architecture.

Important Condition

After startup, it is necessary to check the actual transfer of messages. The presence of a process in the Running state does not guarantee that all required topics are replicated correctly.

It is especially important to evaluate the rules for including and excluding resources.

Replication Policy: Not All Topics Should Be Copied

In a large Kafka cluster, there can be tens or hundreds of topics. For example, some contain critical business events.

Others are auxiliary technical messages. The third are used for temporary testing.

Not all of them necessarily require the same Disaster Recovery.

Business Critical Topics

May require the strictest parameters.

Operational Topics

Require a separate criticality assessment.

Temporary Topics

May not be included in backup replication.

Internal Topics

• Should not be blindly replicated like regular business topics.
• Therefore, the enterprise must define a list of relevant resources.
• For example:

This is a conceptual model.

Actual solutions depend on RPO, RTO, legal obligations, and business needs.

At the same time, it is important to consider that even a technically non-critical topic may contain confidential information.

Therefore, the replication policy must consider not only availability but also Data Governance.

Heartbeats: How to Assess Inter-Cluster Path Availability

MirrorMaker 2 supports service heartbeat messages. They help assess the relevant connectivity between clusters.

For example, an enterprise can monitor whether expected heartbeat events are arriving in the backup environment.

What Heartbeat Shows

It can confirm the operation of a specific replication path and related service components.

What It Does Not Prove

A heartbeat by itself does not guarantee that all business messages have been successfully copied. For example, a specific topic might be incorrectly excluded from the replication rule.

The service heartbeat will continue to arrive.

Therefore, additional checks are required.

Topic-Level Monitoring

The actual replication of critical topics is monitored.

Replication Lag

The corresponding lag is determined.

Data Validation

If necessary, control characteristics of transmitted messages are checked.

Recovery Test

It is necessary to confirm that the backup consumer can correctly resume processing.

Thus, heartbeat is a useful signal, but not a complete criterion for Disaster Recovery readiness.

Consumer Checkpoints: The Most Important Part of Failover

One of the key challenges of cross-cluster failover is the consumer position. For example, a consumer in cluster A has processed messages up to a certain offset.

After a disaster, the enterprise wants to continue operations in cluster B. However, the offsets of two independent Kafka clusters do not necessarily match.

Even if they contain corresponding messages, their positions may differ.

MirrorMaker 2 uses offset syncs and checkpoints mechanisms to maintain corresponding mapping.

Offset Sync

Reflects certain correspondences between the positions of the source and target clusters.

Checkpoint

Contains information about the progress of consumer groups.

Offset Translation

Allows defining an approximate corresponding position in the backup cluster.

Limitations

It cannot be assumed that any arbitrary offset automatically has an exact and accessible match.

KIP-382 provides for corresponding internal topics and mechanisms, but the failover process itself requires coordination.

Practical Risk

If the consumer starts too early, some messages might be processed multiple times. If it starts too late, some events might be missed.

Therefore, idempotency and independent result verification are necessary.

Active–Passive and Active–Active: Different Strategies

For Disaster Recovery, an enterprise can consider several models.

Active–Passive

• The primary cluster receives productive business messages.
• The standby cluster receives corresponding copies and is used after failover.
• For many critical enterprise systems, this model is simpler to control.

Active–Active

• Multiple clusters can serve their own local streams.
• For example, two regions independently create corresponding business events.
• MirrorMaker 2 supports corresponding mechanisms for complex topologies.
• But this does not mean global transactional consistency of all business operations.

Risk Example

• Two regions simultaneously change the same corporate object.
• As a result, different systems may receive conflicting states.
• MirrorMaker 2 will not automatically determine which business decision is correct.
• This requires rules for ownership, routing, and reconciliation.

Model Selection

For a large enterprise, it is important to start with business requirements, not with the most complex topology.

RPO and RTO for Kafka

As with other critical systems, it is necessary to define the Recovery Point Objective and Recovery Time Objective.

RPO

Describes the acceptable lag or loss of information relative to a defined recovery point.

RTO

Defines the target recovery time for an information service. For example, a company might set an RPO target of five minutes for a specific event stream.

However, MirrorMaker 2 uses asynchronous transfer.

Therefore, the actual lag depends on the network, performance, cluster status, and relevant configuration.

Example Requirements

These values are examples. They are not universal recommendations for all businesses.

It is particularly important that for some confirmed financial transactions, even a short-term data loss can be unacceptable.

In such a case, a broader architecture for maintaining an authoritative transactional state is required.

Kafka should not be automatically considered the single source of truth for all business operations.

Split-Brain: How to Avoid Two Independent Active Clusters

One of the dangerous scenarios is a partial network failure. For example, the primary data center becomes unavailable to some users.

The enterprise activates the backup cluster. Later, the connection is restored.

If both clusters simultaneously accept conflicting business operations, a Split-Brain scenario can occur. For example, a single customer object will be modified in both regions.

Appropriate procedures are required for protection.

Fencing

Limiting dangerous parallel writes.

Active Cluster Ownership

Clear definition of the primary cluster for a specific business domain.

Producer Routing

Control of new event routes.

Consumer Routing

Correct connection of relevant consumers.

Recovery Coordination

Defined switching order.

Reconciliation

Verification of the final state after a failure.

MirrorMaker 2 helps to transfer messages, but does not replace these rules.

Therefore, Disaster Recovery must be developed together with business owners.

Kafka, Debezium, and DWH: How to Restore an End-to-End Process

In large enterprises, Kafka is often just an intermediate component. For example, ERP stores the authoritative transactional state.

Debezium captures relevant changes. Kafka transmits messages.

DWH generates analytical results.

If a failure occurs, the entire chain needs to be restored.

ERP

The transactional database’s relevance is checked.

CDC

The relevant log read position is controlled.

MirrorMaker 2

The completeness of transmitted messages is assessed.

Consumers

Verifies the correctness of further processing.

DWH

Necessary facts and measures are restored.

MDM

Historical consistency of corporate identifiers is controlled.

Data Quality

Appropriate checks are performed.

Reconciliation

The result is compared with authoritative sources. For example, the DWH could be restored from a backup at 02:00.

And Kafka contains events up to 02:30.

It is necessary to determine which messages have already been applied and which should be retried. Without idempotency, duplicates can be created.

Therefore, a complete Recovery Runbook must consider all informational components.

Data Quality after Failover

It is especially important not to equate Kafka recovery with the readiness of corporate analytics. For example, a backup Kafka cluster became available.

Consumers continued to receive messages. But the DWH has not yet passed control reconciliations.

In this case, the official analytical showcase should not automatically receive a ready status.

Source Completeness

The completeness of the required initial data is checked.

Duplicate Detection

It is determined whether events have been reapplied.

MDM

Corporate identifiers are checked.

Financial Reconciliation

Checksums are compared.

Historical Integrity

Relevant time intervals are controlled.

Data Freshness

The actual latency is determined.

Certification

After successful checks, the dataset can be approved for a defined use. For example, an operational dashboard may display previous metrics.

However, financial reporting may require full reconciliation.

Therefore, it is necessary to separate Operational Data and Certified Data.

MirrorMaker 2 Monitoring

For production operation, it is necessary to monitor the replication status.

KIP-382 envisages corresponding metrics for latency, record count, message age, and checkpoint time.

It is important not to evaluate the result solely based on the heartbeat’s arrival. For example, it might be running, but a critical topic could be excluded from replication rules.

Therefore, checks of the actual business flow are necessary.

Alerting

Appropriate notifications are configured for critical deviations.

Incident Management

Responsible parties are assigned.

Root Cause Analysis

After an incident, the root cause is determined.

Recovery Testing

Appropriate tests are conducted.

Thus, MirrorMaker 2 should be part of the overall Data Reliability system.

Practical Scenario: Backup Kafka Cluster for a Trading Holding

Let’s consider a conditional enterprise that uses ERP, Debezium, Kafka, and a central DWH. The main data center hosts the production system.

The backup environment is intended for Disaster Recovery.

Stage 1. Business Requirements

Critical topics and corresponding RPO/RTO are defined.

Stage 2. Inventory

Producers, consumers, topics, and information dependencies are documented.

Stage 3. Architecture

Active-Passive is chosen for critical flows.

Stage 4. Backup Cluster

The corresponding Kafka infrastructure is created.

Stage 5. MirrorMaker 2

Replication between clusters is configured.

Stage 6. Topic Policies

The necessary resources and their transfer rules are defined.

Stage 7. Security

Authentication, permissions, and encryption are configured according to requirements.

Stage 8. Heartbeats

Service connectivity is checked.

Stage 9. Checkpoints

Relevant information about consumer groups is monitored.

Stage 10. Monitoring

Actual latency is evaluated.

Stage 11. Failover Testing

The primary cluster’s unavailability is simulated.

Stage 12. Fencing

Checking the correctness of limiting parallel writes.

Stage 13. Consumer Migration

Resuming consumer operations from the appropriate position.

Stage 14. DWH Recovery

Performing necessary event re-application.

Stage 15. Reconciliation

Checking control totals.

Stage 16. Business Acceptance

Relevant owners confirm the readiness of the information service.

Stage 17. Failback

A return to the primary circuit can be planned according to a separate procedure.

Stage 18. Post-Test Review

Results and necessary improvements are documented.

This allows for verification of not only the replication itself but also the correctness of the entire information process.

Failback: Returning to the Primary Cluster

One of the most complex scenarios is returning after the disaster has ended. For example, the enterprise switched to the backup cluster B.

New business events occurred during operation in the backup environment. Later, the primary cluster A was restored.

It is not possible to simply return producers to A without determining what happened during the operation of the backup circuit.

New Events

It is necessary to establish their origin.

Replication Direction

It is necessary to evaluate the corresponding transmission scheme.

Topic Naming

Consider the names of remote topics.

Consumer State

Check the actual positions.

Duplicate Handling

Ensure idempotency.

Reconciliation

Confirm the business result is correct.

For Active-Passive, this procedure typically requires separate planning. For example, an enterprise can temporarily suspend the relevant producer records, complete the necessary synchronization, and resume operation in the main cluster after verification.

However, the specific order depends on business criticality and architecture.

There is no universal MirrorMaker 2 command that will automatically ensure the correct failback of any enterprise application.

Data Governance and GDPR in Cross-Cluster Replication

For enterprises subject to GDPR, it is necessary to consider that Kafka topics may contain personal data. For example, an order message may contain a customer identifier, contact, or address details.

Replication to another data center creates a corresponding information flow.

Purpose Limitation

The purpose of replication must be defined.

Data Minimisation

Personal attributes should not be unnecessarily included in topics.

Access Control

Appropriate access rules are required for the backup cluster.

Data Retention

Message retention periods must be defined.

Security

Cross-cluster transmission must comply with appropriate technical measures.

Cross-Border Transfers

Under certain circumstances, placing a disaster recovery environment in another jurisdiction may require additional legal assessment.

Data Lineage

Relevant data flows must be documented. It is important not to assume that backup or replication automatically exempts from legal obligations regarding personal data.

Similarly, Kafka retention should not be considered a universal procedure for deleting all derivative copies.

This requires a comprehensive Data Lifecycle Management policy.

Economic Assessment of Disaster Recovery

Creating a second Kafka cluster requires additional resources.

For example, an enterprise must consider:

• servers or cloud computing;
• storage systems;
• network channels;
• message queuing;
• administration;
• monitoring;
• testing;
• backup of other information components.

However, the absence of Disaster Recovery also creates risks. For example, prolonged unavailability of event-driven infrastructure can disrupt logistics or related partner services.

Therefore, the economic assessment should compare not only direct costs but also the business consequences of downtime.

Active–Passive

May be appropriate for defined critical processes.

Active–Active

Can provide additional capabilities, but increases complexity.

Selective Replication

Allows selection of critical topics.

Retention

Requires an appropriate balance between cost and Recovery Window.

Capacity Planning

It is necessary to check if the standby cluster can handle the required load. For example, a standby environment with insufficient resources may work normally during test replication, but may not cope with all production consumers after a disaster.

Therefore, appropriate load testing is required.

Intecracy Group Competencies in Cross-Cluster Replication and Data Recovery

Building a robust enterprise event-driven architecture requires several specializations. IQusion IT (iqusion.com) operates in the fields of system integration, electronic document management, and BPM. These areas can be used for restoring application interaction, working with business events, and coordinating relevant response procedures.

Softline IT (softline.kiev.ua) has competencies in server, network infrastructure, and storage systems. They provide the foundation for Kafka clusters, network connectivity, backup, and corresponding Disaster Recovery infrastructure mechanisms. SL Global Service (sgs4business.com) specializes in cloud integration, virtualization, and hybrid infrastructure. Such competencies can be used to organize backup clusters in relevant on-premises or cloud environments.

At the same time, RPO, RTO, and business readiness criteria must be approved by the enterprise according to the criticality of information processes.

Practical Implementation Guidelines

Apache Kafka 2.4, released in December 2019, marked a significant milestone in the development of enterprise event-driven infrastructure. MirrorMaker 2 introduces new inter-cluster replication capabilities based on Kafka Connect.

It allows for message transfer between independent clusters, maintaining corresponding checkpoints, and utilizing heartbeat mechanisms for connectivity control. For businesses, this opens up opportunities for building geographically distributed infrastructure and Disaster Recovery.

However, MirrorMaker 2 does not automatically solve all business consistency tasks. It is necessary to correctly define the topology, replicated topics, access rights, Consumer Recovery, Failover, and Failback.

Idempotency and controlled data recovery in DWH play a special role. After a failure, it is not enough for a company to simply launch a backup Kafka cluster.

It is necessary to confirm that business events are correctly received, processed, and reflected in corporate information systems.

The main principle of Disaster Recovery for Apache Kafka is that replication should ensure not only the technical copying of messages between clusters but also the predictable recovery of the entire information service – with defined RPO, RTO, correct consumer positions, duplicate control, and verified business results.