Disaster Recovery / Business Continuity / Data Integrity
Why a Backup Data Center Doesn’t Guarantee Business Resumption
Most large enterprises have certain mechanisms in place to protect their information systems from failures.
Backup servers, mirrored storage systems, network channels, and regular backups are utilized.
For critical databases, replicas may be maintained at another site. However, the presence of appropriate hardware does not yet mean that an enterprise can quickly resume operations after a large-scale disruption.
For example, a corporate ERP might have an up-to-date backup database but lack a ready application server.
Another scenario is that servers and databases are restored, but the system cannot operate due to the unavailability of Active Directory, DNS, or an external integration service.
A situation is also possible where, after a disaster, the enterprise successfully launches a backup ERP, but some business operations remain in other information systems that were not synchronized up to the same point in time.
As a result, discrepancies arise between orders, contracts, inventory levels, and financial documents.
Therefore, disaster recovery should be considered not as a separate server administration function, but as a comprehensive process of restoring corporate information services and related business operations after a significant disruption of their work.
For this purpose, Disaster Recovery Planning is used.
It should be a component of a broader Business Continuity Management system.
Business Continuity and Disaster Recovery
The concept of Business Continuity encompasses an organization’s ability to continue critical activities during and after disruptions to normal operating conditions.
Disaster Recovery primarily focuses on restoring information systems, technological infrastructure, and data. These functions are related, but not identical.
For example, a trading company might lose access to its primary ERP.
During recovery, some orders may be taken through a backup procedure and subsequently entered into the system.
This is an organizational business continuity measure.
Meanwhile, the IT department is restoring servers, databases, and integration channels.
This is a component of Disaster Recovery.
For systemic business continuity management, ISO 22301:2012 can be used — a standard published in May 2012, which establishes requirements for a business continuity management system.
Importantly, the standard does not define a universal RTO for ERP or a mandatory replication technology.
The enterprise must independently determine its critical processes and the acceptable consequences of disruptions.
Business Impact Analysis: where the recovery plan begins
One of the first stages is the Business Impact Analysis (BIA). Its purpose is to determine which business processes are most dependent on information systems and what the consequences of their unavailability will be.
For example, an enterprise may determine that the inability to process shipments for four hours will result in significant financial losses.
At the same time, the unavailability of the monthly analytical data mart for the same period may be acceptable.
Therefore, the corresponding systems require different levels of redundancy.
Dependency Analysis
For each process, it is necessary to identify the information services without which it cannot operate.
For example, order processing may depend on ERP, MDM, an authentication system, and an integration API.
If only the ERP database is backed up, but MDM is not restored, the process may remain blocked.
Prioritization
Restoration should be performed in a sequence that aligns with business needs.
For example, authentication services, network infrastructure, and the core DBMS are restored first.
After that, ERP, integration services, and custom applications are launched.
However, the specific order depends on the architecture.
In some systems, application components can be launched in parallel if dependency waiting is implemented correctly.
Main Disaster Recovery Scenarios
Several disaster recovery architectural models are available for enterprises.
They differ in cost, readiness of the backup environment, and potential recovery time. From Backups to Continuous Replication
Conceptual Comparison
Backup & Restore
Resources and data are recovered after an incident
Pilot Light
Part of the critical components are maintained in advance
Warm Standby
A reduced backup environment runs constantly
Active–Active
Multiple sites serve the load simultaneously
Schematically shows general growth in complexity and constant costs, not measured values. Actual RTO depends on architecture and test results.
Backup & Restore
The simplest approach involves restoring infrastructure and data from backups.
It can be economically feasible for auxiliary systems that allow for significant downtime.
However, recovery can take hours or longer depending on the amount of data.
Pilot Light
A portion of critical components is maintained in a standby environment.
For example, a database may be replicated, while application servers are only started after a disaster.
This reduces the need for constant resources but increases recovery time compared to a fully ready site.
Warm Standby
The standby infrastructure runs constantly in a reduced configuration.
After a disaster, its resources are scaled up to the required level.
This can be a practical compromise between cost and recovery speed.
Active–Active
Multiple sites simultaneously process a portion of the production workload.
This approach can increase availability but creates significantly more complex requirements for state distribution and transactional consistency.
This is especially true for business operations that should not be executed independently on two nodes without proper coordination.
These scenarios were already used in cloud DR practices in 2016–2017.
Backup: The First Line of Defense
Regardless of the chosen strategy, a business needs backups. Replication does not replace them, as erroneous changes can be transmitted along with correct ones.
For example, an administrator accidentally deletes an important table. In many configurations, this operation will be replicated on the backup server.
Recovery requires an independent copy of the previous state.
Full Backups
Contain a defined full set of data.
Differential Backups
For supported DBMS, they allow storing changes relative to the corresponding base backup.
Transaction Logs
Allow restoring the sequence of committed operations according to the capabilities of the specific platform. For critical systems, it is necessary to define the frequency of backups and storage on independent resources.
It is also important to verify the integrity of backup files and perform practical recovery tests.
A successful backup completion message does not guarantee that the entire information system can be restored.
Point-in-Time Recovery: Recovery to the Point of Failure
One of the most important features of enterprise DBMS is Point-in-Time Recovery (PITR). It allows restoring the database state to a specified moment in time, provided that the necessary base copies and logs are available.
For example, an incorrect mass update was performed at 14:35. If the system has a proper backup chain, it is possible to estimate recovery to the moment immediately before the error.
However, it must be considered that correct operations that occurred after the selected moment will also be absent.
Therefore, restoring a production DBMS may require business alignment.
PostgreSQL 9.6
PostgreSQL supports PITR through base physical backups and WAL archiving. The pg_basebackup utility is used to create a base backup.
It can serve as the basis for disaster recovery or for creating a standby server. It is important to note that a logical pg_dump is not a physical base backup for WAL replay.
These are different mechanisms.
Microsoft SQL Server
SQL Server supports recovery to a specific point in time with the correct recovery model configuration and the availability of the necessary transaction log chain. This is achieved using RESTORE LOG with the STOPAT parameter.
Restoration must be performed in the correct sequence of relevant backups.
Oracle Database
Oracle RMAN allows for the implementation of relevant recovery scenarios using backups and archived logs.
The specific method depends on the database configuration and recovery requirements.
Consistency Across Multiple Enterprise Systems
One of the most complex problems in Disaster Recovery is the restoration of not just a single database, but the entire information chain. Let’s consider an enterprise that uses ERP, CRM, and a document management system.
At 10:15, a client created an order in CRM. At 10:16, ERP processed the corresponding operation.
At 10:17, the DMS generated a document for approval.
At 10:20, a disaster occurred at the central data center. Suppose CRM is restored to 10:19, ERP to 10:14, and DMS to 10:18.
As a result, an inconsistent state may arise. CRM contains the order, ERP is unaware of it, and DMS has a related document.
Therefore, the individually successful recovery of each system does not yet guarantee the consistent recovery of the corporate business process.
Consistent Recovery Point
For interconnected systems, it is important to determine if they can be restored to compatible states. This does not always mean an absolutely identical timestamp.
In a distributed environment, it is necessary to consider effectively confirmed operations and inter-system messages.
Reconciliation
Reconciliations may be performed after recovery. For example, CRM and ERP orders are compared.
Discrepancies found are passed to a controlled information recovery procedure.
Replaying Events
If an enterprise uses Kafka or another messaging platform, some integration events can be reprocessed if the necessary history is available.
However, idempotency must be ensured so that repeated execution does not create duplicate documents.
Business Responsibility
For financial operations, adjustment decisions must be made in accordance with established rules.
Automatic re-creation of a missing record is not always correct, especially if it has legally significant consequences.
Disaster Recovery for DWH and Data Lake
Analytical systems have different requirements than transactional ERPs. If the DWH is formed from operational sources, some information can theoretically be reloaded.
However, this is only possible if the sources, historical logs, and transformation rules are preserved.
DWH Recovery
The enterprise must determine if all necessary historical data is available in the ERP. For example, if the source only stores the last two years, and the DWH stores a ten-year history, a full reload may not be possible.
Therefore, the analytical storage also requires independent backup.
Data Lake Recovery
For HDFS, block replication and corresponding backup procedures can be used.
However, replication between nodes of the same cluster does not guarantee protection against the loss of an entire data center.
For critical datasets, independent copies or cross-cluster transfer should be evaluated.
Metadata
It is particularly important to store catalogs, Hive structures, ETL descriptions, and other metadata. If, after a failure, the enterprise has the physical files but has lost information about their content and transformations, the analytical system may remain unusable.
Therefore, recovery must cover both data and metadata.
Cloud Infrastructure for Disaster Recovery
In 2017, enterprises can use AWS and Microsoft Azure as additional sites. For example, a local system can create backups and then transfer them to external storage.
Another option is to maintain virtual machines or databases in a cloud environment.
Advantages
The cloud model allows resources to be obtained without the need to build a full-fledged second physical data center.
For certain scenarios, this can reduce initial capital expenditures.
Limitations
Restoring a large database depends on channel bandwidth, backup availability, and resource startup time.
Storage and data transfer costs must also be considered.
Vendor Lock-in
The loss of availability of a specific cloud service can also affect the backup scenario. Therefore, it is necessary to determine which failures the chosen strategy should cover.
For example, a backup site in the same availability zone does not provide the same level of geographical independence as another region.
However, cross-region migration raises additional issues of latency, security, and cost.
Disaster Recovery Runbook: A Clear Procedure for Responsible Teams
One of the most important practical documents is the Disaster Recovery Runbook.
It should contain a sequence of actions after a failure.
For example, for a corporate ERP, it is necessary to define:
- Who declares the disaster mode and makes the decision to switch.
- How the status of the primary and backup sites is checked.
- Which nodes need to be isolated to prevent split-brain.
- How is the suitability of a backup or replica confirmed.
- How are the database, application servers, and integrations launched.
- How are user and business operations verified.
- Who approves the completion of disaster recovery.
This procedure must be reviewed regularly. For example, if the company has changed IP addresses, the authentication system, or the integration mechanism, the previous Runbook may become obsolete.
It is also important to ensure the availability of documentation during the unavailability of the main corporate network.
Disaster Recovery Testing
One of the most common problems is the lack of practical testing. A company may spend significant funds on backup infrastructure, but perform a full switchover for the first time only during a real disaster.
In such a case, unexpected dependencies may be revealed.
For example, a backup server has an up-to-date database but uses an outdated certificate.
Or an application service cannot connect due to incorrect DNS parameters.
Therefore, regular testing is necessary.
Backup Restoration Test
The ability to restore data on a separate server is checked.
Failover Test
The failure of the primary node is simulated.
The time to switch to the backup system is measured.
Site Isolation Test
The behavior after losing network connectivity is checked.
It is especially important to prevent split-brain.
End-to-End Business Test
After the backup environment is launched, a typical operation is performed.
For example, creating an order, checking the balance, and generating a document.
This is the test that allows us to determine if the business process has truly been restored.
Failback
It is necessary to separately test the return to the primary site. After emergency operation on the backup server, new information may have accumulated.
The return process must ensure its proper transfer and reconciliation.
Assessment of Actual Disaster Readiness
It is advisable for an enterprise to maintain measurable metrics.
| Metric | What is monitored |
|---|---|
| Backup Success Rate | Proportion of successful backups |
| Restore Test Success | Results of test restores |
| Replication Lag | Backup lag |
| Actual RPO | Actual amount of potentially lost information |
| Actual RTO | Recovery time during testing |
| Runbook Accuracy | Procedure relevance |
| Business Test Success | Ability to perform critical operations |
| Failback Readiness | Readiness for failback |
| Dependency Coverage | Completeness of dependent system consideration |
It is important to evaluate test results themselves, not just the passport characteristics of the equipment.
For example, the presence of a synchronous replica does not guarantee an RTO of five minutes if the application ERP requires lengthy manual reconfiguration.
Similarly, fast server startup does not guarantee data completeness.
Practical Scenario: Restoring Operations of a Distribution Company
Consider a hypothetical company that uses ERP, CRM, DMS, and a corporate DWH.
The main data center is located at the head office.
The backup data center is at a different site.
As a result of a failure, the main data center becomes unavailable.
Stage 1: Incident Declaration
Responsible individuals determine the scope of the disruption and activate the disaster recovery plan.
Stage 2: Isolation
The team checks the status of critical nodes and ensures the absence of dangerous parallel operations.
Stage 3: DBMS Recovery
Switching or restoring backup databases according to their technologies is in progress.
Fourth stage: application systems launch
ERP, authentication services, and necessary network components are being restored.
Fifth stage: integrations
The team checks APIs, message queues, and exchange with CRM and EDMS.
Sixth stage: reconciliation
Critical business operations and corporate identifiers are compared.
Seventh stage: business verification
Employees create a test order, perform planned control operations, and check document generation.
Eighth stage: emergency mode operation
The enterprise documents new operations and defines rules for returning to the primary site.
After the incident is completed, the results are analyzed, and the plan is improved.
Thus, Disaster Recovery covers the entire information process, not just server recovery.
Data Governance and Information Responsibility After a Disaster
It is important to determine who has the right to confirm the correctness of the restored information.
The DBA can establish that the DBMS is running and all necessary files are available. However, this does not automatically mean that the restored financial indicators correspond to business operations.
Therefore, for critical information domains, the participation of the Data Owner is necessary. For example, the finance department confirms the correspondence of balances and processed documents.
The Data Steward can coordinate the check for duplicates and conflicts after re-synchronization. The integration team checks intersystem messages.
The Data Architect determines which information dependencies require further analysis.
This distribution allows for the separation of technical recovery from the business acceptance of the result.
Intecracy Group Competencies in Disaster Recovery
Disaster recovery planning requires a combination of various specialized competencies. IQusion IT specializes in system integration, electronic document management, and BPM. These areas are significant for the recovery of application connections and end-to-end business processes.
Softline IT has competencies in server and network infrastructure, storage systems, and corporate infrastructure projects. They are directly related to backup, server recovery, and infrastructure fault tolerance.
At the same time, recovery priorities and acceptable losses should be determined by the company’s management according to the business impact.
Practical Guidelines for Implementation
Disaster Recovery is an important component of corporate information risk management.
For effective disaster recovery, having a backup server or a database replica is not enough. It is necessary to identify critical business processes, approve RPO and RTO, create a consistent backup architecture, and prepare recovery procedures.
Independent backups, Point-in-Time Recovery capabilities, and inter-system consistency control are of particular importance. For large enterprises, it is important to consider dependencies between ERP, CRM, ECM, DWH, MDM, and integration platforms.
Disaster recovery plans must be regularly tested in practice.
In this case, the ultimate criterion of success is not the launch of the DBMS, but the restoration of business operations with reliable information.
Effective Disaster Recovery is the ability of an enterprise, after a major incident, to restore critical information systems, reconcile corporate data, and return to managed operations within predefined business requirements.
Author