Keeping analytics running during remote work

Business Continuity / Data Management / Remote Work / Enterprise Architecture

New working conditions for businesses require a review of the information strategy

March 2020 presented businesses with challenges that only a few months ago were considered primarily as crisis planning scenarios.

On March 11, the World Health Organization characterized the spread of COVID-19 as a pandemic. At the same time, restrictive measures began to be introduced in various countries, affecting people’s movement, office operations, transport, logistics, and the organization of production processes.

For many businesses, this means the need to quickly change familiar ways of working.

Some employees are transitioning to remote interaction. Individual departments are working on altered schedules.

Suppliers and partners may temporarily change the order of fulfilling their obligations.

In such conditions, the availability of reliable corporate information becomes particularly important. For example, a trading holding has several ERPs, a centralized CRM, and a corporate Data Warehouse.

The usual model assumes that managers work in the office daily, use the internal network, and receive reports through a local BI platform. However, some employees are forced to work remotely.

Questions immediately arise. Are critical analytical resources available outside the office?

Can the finance department complete reporting without the physical presence of all employees? Will ETL processes work if administrators cannot promptly come to the data center?

Can the company identify and correct an information error if the Data Steward, DBA, and business owners are in different locations? These issues are not solely related to remote work.

They demonstrate a broader problem: the dependence of the corporate information system on physical location, individuals, and manual procedures. Therefore, businesses need to review Business Continuity Planning (BCP) – a set of organizational and technical measures aimed at continuing critical business functions during adverse events.

For Data Management, the main task becomes ensuring the availability, timeliness, and correctness of corporate data, even with significant changes to the normal operating mode.

Business Continuity and Disaster Recovery – Different Concepts

One common mistake is equating Business Continuity Planning with server backups. Disaster Recovery is indeed an important component of resilience.

However, it does not cover all organizational aspects.

Disaster Recovery

Focused on restoring relevant information systems after a technical failure.

For example, server hardware failure or unavailability of the main data center.

Business Continuity

Has a broader purpose. For example, the information infrastructure may be working correctly, but the enterprise may not be able to complete a critical process due to the absence of authorized employees.

Or due to the temporary unavailability of an external data provider.

Thus, for corporate analytics, it is necessary to plan not only for technical recovery but also for maintaining the necessary information service.

For example, if a daily financial report is critical for liquidity management, the company must know how to ensure its preparation in the absence of the usual team.

Defining Critical Information Services

The first practical step is to inventory information processes. For a large enterprise, this could be hundreds of ETLs, APIs, reports, and related analytical resources.

Not all of them are equally critical. For example, a market research report can be postponed.

However, daily accounts receivable monitoring may remain necessary.

Therefore, it is advisable to set priorities.

This is a conditional example.

For a specific enterprise, priorities should be determined by the respective business owners.

Business Impact Analysis

It is necessary to assess the consequences of the unavailability of each process.

Dependencies

Establish which systems and employees it depends on.

Recovery Requirements

Define acceptable latency and recovery time.

Alternative Procedures

Provide appropriate temporary workarounds. It is especially important not to create equally complex backup architectures for all data sets.

Resources should be allocated according to actual business risk.

Remote Work and Secure Access to Analytics

In normal operation, many corporate systems are only accessible from the internal network. For remote work, appropriate controlled access must be provided.

For example, a financial analyst must work with the BI platform and necessary reports. And a Data Engineer must check ETL and perform relevant maintenance operations.

However, exposing internal databases directly to the internet is not a proper solution to the problem. This requires an appropriate architecture.

Remote employee

Managed device · proper authentication

Secure access channel

VPN / corporate gateway / appropriate policies

Access control

Identification · authorization · auditing

BI / Reports

ETL Monitoring

BPM / DBMS

Conceptual model of remote access. Direct user access to production DBMS is not a mandatory condition for remote analytics.

Authentication

It is important to correctly verify the user’s identity.

For critical resources, multi-factor authentication can be used.

Authorization

An employee should only receive the necessary permissions.

Endpoint Security

The state of the device from which access is made must be taken into account.

Logging

Relevant critical events must be monitored.

Privileged Access

Administrative privileges should have separate rules.

Network Capacity

It is necessary to assess whether the infrastructure can support a significantly larger number of remote users. For example, a VPN gateway designed for dozens of simultaneous connections may be insufficient if hundreds of employees work remotely.

Therefore, the technical solution must include appropriate load testing.

BI Availability Does Not Mean Data Availability

For corporate analytics, it is important to distinguish between interface availability and information relevance. For example, an employee may successfully open a financial report through a secure channel.

However, if ETL did not run from the previous day, the result will be outdated.

Therefore, it is necessary to monitor several independent characteristics.

BI Availability

Is the relevant analytical system available?

Data Freshness

When was the information last updated?

Data Completeness

Have all mandatory sources been received?

Data Quality

Have the necessary checks passed?

Certified Status

Is the dataset suitable for a specific use case?

For example, for the morning financial dashboard, you can define:

Such requirements should be known to users. For example, if a report contains preliminary data, this must be clearly indicated.

Do not present an incomplete result as final.

DataOps: Reducing Reliance on Manual Work

One of the main risks to Business Continuity is the dependence of information processes on individual employees. For example, a company has a critical ETL that is run manually by only one specialist.

If this person is temporarily unavailable, the process may stop. Another case is when only one DBA knows the recovery procedure for a specific table.

To reduce such risks, DataOps should be used.

Version Control

SQL, Python, configurations, and DAGs should have a controlled history.

Orchestration

Scheduled processes must run without requiring daily manual initiation.

Automated Testing

Structures, data, and business results need to be validated.

Documentation

Critical processes must have a clear description.

Release Management

Updates are performed according to defined procedures.

Monitoring

Errors must be detected without constant manual interface checks.

Recovery

Required verified recovery instructions.

For example, if a Data Engineer is unavailable, another authorized specialist should be able to understand the relevant DAG and determine the safe restart order.

This is why documentation and automation are components of Business Continuity.

Apache Airflow 1.10.9: Managed Orchestration

As of March 2020, Apache Airflow 1.10.9 is available.

It was released in February as a bugfix release of the 1.10 branch.

The next Airflow 1.10.10 will only be released in April 2020.

For corporate BCP, Airflow can be used to automate the execution of dependent information tasks.

For example:

ERP Extract
    ↓
Schema Validation
    ↓
MDM Matching
    ↓
DWH Load
    ↓
Reconciliation
    ↓
Certified Publication

This sequence helps reduce the risk of accidental policy violations.

Retries

Can be used for temporary technical failures.

SLA Monitoring

Allows detection of certain deadline violations.

DAG Logs

Help analyze relevant errors.

Backfill

Can be used to re-run defined historical periods.

However, it is important not to overestimate the tool.

Airflow does not automatically determine the correctness of financial indicators.

Similarly, a restart is not always safe.

If ETL does not support idempotency, re-execution may create duplicates.

Therefore, Business Continuity must consider both data quality and the appropriate transactional model.

Apache NiFi 1.11.3: Managing Integration Flows

Apache NiFi can be used for enterprise information flows.

As of the date of this article, NiFi 1.11.3 is available, released on February 26, 2020.

The release includes fixes for resource leaks and improvements to relevant integration components.

For Business Continuity, queue control capabilities, Data Provenance, and parameterization are of particular importance.

Queues

Allow separation of relevant transmission stages.

Backpressure

Helps to limit excessive information accumulation.

Provenance

Allows analysis of the history of relevant FlowFiles within the available period.

Parameter Contexts

Available since NiFi 1.10 and can simplify configuration management.

Monitoring

Required for detecting queue overflows and technical violations.

However, it should not be assumed that saving the NiFi configuration automatically implies the possibility of a full recovery of all unfinished FlowFiles after a failure.

To achieve this, the state of relevant repositories and other operational characteristics must be considered.

Appropriate backups and verified procedures are also required.

Dependency on External Sources

Business Continuity for Data Management must consider not only its own infrastructure.

For example, an enterprise may receive data from a logistics partner or an external payment system.

If such a provider is temporarily unavailable, the internal ETL will not be able to obtain the relevant information.

Therefore, the behavior of the analytical system must be defined.

Last Confirmed State

Under the right conditions, the last verified set can be used.

Freshness Warning

Users should see that the information is outdated.

Partial Readiness

Data from available sources can be displayed separately, provided that the limitations are correctly indicated.

Deferred Loading

After the source is restored, the necessary Backfill is performed.

Reconciliation

It is necessary to check if all records have been received.

For example, if a partner did not provide data for two days, it is not enough to simply get the last file after restoration.

It is necessary to check the completeness of the entire missed period.

Therefore, critical integrations require appropriate information contracts.

RPO and RTO for Analytical Services

For Business Continuity, target recovery metrics must be defined.

Recovery Point Objective

Describes the acceptable data loss point in case of a disaster.

Recovery Time Objective

Defines the target service recovery time.

However, Data Management requires additional characteristics.

For example, after recovery, the DWH must not only be available but also correctly populated.

These are just examples.

Actual values should be determined based on business risk and infrastructure capabilities.

It is especially important to verify their execution in practice.

For example, the presence of a backup does not guarantee that the DBMS can be restored in a timely manner.

Therefore, appropriate Recovery Tests are necessary.

Business Continuity for Data Stewardship

One of the less obvious problems is the dependence of quality control on responsible specialists.

For example, MDM daily identifies several ambiguous client matches.

Usually, one Data Steward checks them.

However, during the temporary absence of an employee, corresponding exceptions can accumulate.

Therefore, it is necessary to organize replacements.

Authorization Distribution

It is necessary to define who can make relevant decisions.

Documented Rules

Matching and Survivorship must have a clear methodology.

BPM / Workflow

For complex cases, appropriate approval processes can be used.

Audit Trail

Critical decisions must be documented.

Escalation

Under defined conditions, the exception is transferred to another authorized specialist.

Remote Access

Employees must have a secure way to work with relevant tools remotely.

Thus, Data Stewardship should be a corporate process, not an informal function of one person.

Information Security During Remote Work

The rapid transition to remote interaction methods creates additional risks. For example, employees might start transferring corporate Excel files via personal mailboxes or external services not provided for by the company’s information policy.

This may be unacceptable for critical datasets.

Corporate Channels

Properly defined transfer methods must be used.

Access Control

Permissions must correspond to the user’s role.

Data Minimisation

Full tables should not be exported if only an aggregated report is needed.

Endpoint Protection

The status of devices must be considered.

Audit

It is important to monitor relevant critical actions.

Backup

It is necessary to properly organize the storage of work documents.

For companies subject to GDPR, these measures must also take into account relevant legal obligations.

However, remote work in itself does not constitute a GDPR violation.

The actual purposes, processing methods, and relevant security measures are important.

Practical Scenario: BCP for a Corporate Data Warehouse

Let’s consider a hypothetical holding company with multiple ERPs, a centralized CRM, and a DWH.

The company is forced to transition a significant portion of its employees to remote work.

Phase 1. Business Criticality

Required information services are identified.

Phase 2. Data Ownership

Responsible parties are assigned for each critical dataset.

Phase 3. Remote Access

Secure access is configured.

Phase 4. Network Capacity

The throughput of relevant channels is checked.

Phase 5. ETL Inventory

Critical information processes are documented.

Stage 6. Airflow / Orchestration

Scheduled execution is automated.

Stage 7. Data Quality

Appropriate control checks are established.

Stage 8. Data Steward Backup

Authorized deputies are identified.

Stage 9. Monitoring

Remote monitoring is configured.

Stage 10. Incident Management

The response procedure is documented.

Stage 11. Recovery Testing

Recovery of necessary information services is being checked.

Stage 12. External Dependencies

Risks of data provider unavailability are assessed.

Stage 13. Business Communication

Users receive information about the status of relevant datasets.

Stage 14. Review

The plan is updated based on practical work results.

This model helps reduce the dependence of corporate analytics on a standard office environment.

KPI Business Continuity for Data Management

It is important to evaluate all characteristics together. For example, successful user connection via VPN does not guarantee that they are receiving up-to-date data.

Similarly, a functional DWH may be unsuitable for regulated reporting if control reconciliations are not completed.

Intecracy Group Competencies in Business Continuity

Ensuring the continuity of corporate information processes requires several technological specializations. Data Management IG (dmig.com.ua) specializes in ETL, MDM, Data Quality, DWH, and corporate analytics. These areas are important for maintaining information pipelines, quality control, historical data, and the correctness of analytical results under non-standard operating conditions.

IQusion IT (iqusion.com) operates in the fields of system integration, electronic document management, and BPM. These competencies can be used to automate remote business processes, approvals, document flow, and handling information exceptions. Softline IT (softline.kiev.ua) has competencies in server, network infrastructure, and storage systems. They provide the foundation for remote access, DBMS performance, backup, and relevant Disaster Recovery mechanisms.

SL Global Service (sgs4business.com) specializes in cloud integration, virtualization, and hybrid infrastructure. These areas can be used to create backup environments and controlled remote accessibility of relevant services.

At the same time, business priorities, acceptable downtime periods, and information correctness criteria must be determined by the enterprise.

Practical Guidelines for Implementation

The events of March 2020 highlight the importance of Business Continuity for corporate information systems.

The COVID-19 pandemic and related changes in work organization are forcing companies to review their dependence on the physical presence of employees for critical processes. For Data Management, this means ensuring the availability of analytics, automated ETL execution, Data Quality control, and proper recovery procedures.

Apache Airflow 1.10.9 and NiFi 1.11.3 can be used as components of a corresponding technical architecture.

However, software tools do not replace organizational responsibility.

Successful BCP requires Data Ownership, documented Runbooks, prepared substitutes, and appropriate Incident Management procedures.

Control of Data Freshness and information accuracy is of particular importance.

The main principle of Business Continuity for Data Management is that the company must maintain the ability to receive timely and reliable corporate data regardless of employee location or temporary restrictions affecting normal work organization.