Checking data after business processes change

Data Quality / Data Reliability / DataOps / Enterprise Data Warehouse

When Yesterday’s Rules No Longer Describe Today’s Business

Corporate information systems are built on certain business assumptions. For example, ERP uses an established order processing procedure. CRM supports a defined customer interaction model. ETL transforms operational records into analytical metrics, and Data Warehouse generates consolidated reporting.

In stable conditions, such rules can work for years.

However, March 2020 demonstrates how quickly business processes can change. On March 11, the World Health Organization characterized COVID-19 as a pandemic. Many countries began implementing measures affecting people’s movement, work organization, logistics, and economic activity.

For businesses, this creates not only organizational but also informational challenges. For example, a trading company starts using additional order reception channels.

Some deliveries are delayed. Individual orders are canceled or postponed.

New statuses for customer inquiries appear in the CRM. Financial departments review relevant forecasts and scenarios.

In such a situation, the Data Warehouse may continue to operate without technical errors, but no longer fully reflect the real business.

For example, the ETL previously considered three order statuses:

NEW, CONFIRMED, CANCELLED.

After changes to the business process, the ERP introduces:

ON_HOLD, PARTIALLY_FULFILLED, RESCHEDULED.

If the corresponding SQL procedures are not updated, analytics may incorrectly determine the number of completed operations. Another example is a company that uses demand forecasting generated from historical data.

Previously, demand had a relatively stable seasonal pattern. Now, the order structure is changing rapidly.

Even a technically correct model can generate inaccurate forecasts because its initial assumptions no longer match the actual situation. Therefore, companies need to review their approach to corporate data control.

Data Quality should ensure not only the correctness of records according to previous rules but also the timely detection of situations where the business rules themselves no longer correspond to real processes.

This requires a combination of Data Governance, automated profiling, DataOps, information contracts, and Data Reliability.

Data Quality and the Change of Business Semantics

Traditional checks often focus on structure and formats.

For example, the amount field must be numeric.

customer_id must not be empty.

The document date must conform to a defined format. Such rules remain necessary.

However, they do not allow for the detection of all changes in business content.

Let’s consider an order:

{
  "orderId": "ORD-2020-5824",
  "customerId": "CUST-501",
  "amount": 1250.00,
  "status": "ON_HOLD"
}

The message may be technically correct.

However, if the ETL does not know the ON_HOLD status, it may incorrectly classify the operation.

Structural Validation

Checks field names and types.

Semantic Validation

Checks their business meaning.

Business Rule Validation

Controls relevant usage rules.

Distribution Monitoring

Helps identify unusual changes in information characteristics.

Data Owner Review

Ensures approval of new definitions.

Thus, the enterprise needs to control not only Schema Drift but also Semantic Drift.

Schema Drift and Semantic Drift: Different Risk Sources

For practical DataOps, it is useful to separate types of changes.

For example, if the number of orders sharply decreases, it does not necessarily mean a technical error. Perhaps the reason is a real change in demand.

Another case is when the number of records decreases due to integration malfunction. Both situations can look similar in the monitoring system.

Therefore, automated checks should help identify deviations, but the final interpretation requires business context. It is especially important not to create a system that automatically considers every significant deviation an error.

During significant changes in business processes, historical norms may temporarily lose relevance.

Data Profiling: a new role for distribution analysis

Data Profiling can be used to detect atypical changes.

For example, the company analyzes order characteristics daily.

Row Count

Number of records received.

Null Rate

Proportion of empty values.

Distinct Values

Number of distinct statuses or categories.

Value Distribution

Proportion of records for each status.

Numeric Statistics

Minimum, maximum, and average values.

Temporal Distribution

Distribution by dates and times. For example, before March, the share of canceled orders was conditionally 3-5%.

After the business conditions changed, it increased to 20%. Such a result can be an important signal for management.

However, one should not automatically block all records as incorrect.

It is necessary to establish whether the change corresponds to actual business events.

Practical Model

Data Profiling can generate automatic alerts. After this, the responsible Data Steward or business analyst assesses the cause.

If the change is justified, the relevant control thresholds may be revised.

It is important to document such decisions.

Why historical thresholds may stop working

Let’s consider an enterprise that uses the rule:

If the number of daily orders deviates from the average by more than 30%, a Data Quality Incident is created.

Under stable conditions, this can be useful. However, during a sharp change in business activity, the system may generate numerous warnings.

For example, the enterprise temporarily ceased operations of some retail outlets. The number of orders decreased.

This is a real business event, not necessarily an information error.

• Therefore, it is advisable to distinguish:
• Data Incident - a confirmed breach of data integrity or availability.
• Business Anomaly - an unusual but potentially real change in business metrics.
• Expected Process Change - an approved change in a business process.

Recommended Approach

Different types of notifications should be used.

For example:

This allows reducing the number of false critical alerts.

Information contracts must change with the business

For corporate integration, it is important to have defined Data Contracts. For example, ERP transmits orders to DWH.

The contract should describe:

• mandatory fields;
• data types;
• allowed statuses;
• identifiers;
• temporal semantics;
• responsibility for changes;
• compatibility rules.

During sharp changes in business processes, contracts may require updates.

For example, an enterprise introduces a new status RESCHEDULED.

Stage 1. Business Definition

The Data Owner defines its meaning.

Stage 2. Contract Update

The relevant API description or message is updated.

Stage 3. Compatibility Test

Existing consumers are checked.

Stage 4. ETL Logic

Transformations are changed as needed.

Stage 5. Regression Testing

The impact on metrics is controlled.

Stage 6. Production Release

Deployment follows a defined procedure.

Stage 7. Monitoring

Actual results are checked.

It is necessary to avoid situations where the ERP team independently changes the status value without notifying the relevant consumers.

This is especially dangerous for corporate financial reporting.

DataOps During Rapid Changes

Under normal circumstances, an enterprise can plan ETL changes for several weeks. However, during a sharp restructuring of business processes, individual information requirements may arise much faster.

For example, management needs a new report on deferred orders. At the same time, it is important not to sacrifice quality control for the sake of speed.

This requires DataOps.

• Business Change
• New Status, Channel, or Rule
• Data Contract
• Clarification of Structure and Semantics
• Git / Pull Request
• Controlled ETL Change
• Automated Tests
• Structural and Business Checks
• Regression Testing
• Impact on Existing KPIs
• Data Owner Approval
• Critical Methodology Approval
• Production Deployment
• Controlled Release
• Monitoring
• Freshness · Quality · Reconciliation

DataOps allows accelerating changes without abandoning business responsibility and relevant control procedures. For example, a new SQL procedure can be implemented quickly if it passes the relevant tests.

However, changing financial methodology still requires proper approval.

Therefore, it is important to distinguish between technical and business risks.

Apache Airflow 1.10.9: Controlled Rollout of New ETLs

For orchestrating information processes, Apache Airflow 1.10.9 is available as of March 2020.

Version 1.10.10 is not yet released—its release will take place on April 9, 2020.

Airflow can help organize dependencies, retries, and deadlines.

For example, updating a financial data mart should start after the necessary MDM processes are completed.

DAG Dependencies

Define the execution order.

Retries

Used for temporary technical errors.

SLA Monitoring

Helps monitor relevant time criteria.

Backfill

Can be used to reprocess defined periods.

Logs

Allow analysis of relevant technical violations.

However, it is important not to try to fix all problems through retries.

For example, if the ERP sends a new unknown status, re-running the ETL will not create the correct business methodology.

In such a case, an update to the relevant contract is required.

Therefore, the Retry Policy must consider the type of error.

Debezium 1.1: CDC development in March 2020

On March 23, 2020, Debezium 1.1.0.Final was released.

This is a new release of the Change Data Capture platform.

It is already available as of the date of this article.

Prior to this, on March 11, the corresponding release candidate 1.1.0.CR1 was issued.

For enterprises, CDC can be important during changes in information processes.

For example, ERP actively updates order statuses.

Instead of fully re-reading a large table, corresponding incremental changes can be transmitted.

However, this creates new requirements for Data Quality.

Schema Evolution

New fields and statuses must be handled correctly.

Source Position

The log reading position needs to be controlled.

Event Deduplication

Duplicate messages should not create duplicates.

Business Semantics

A technical table change does not necessarily constitute a completed business operation.

Reconciliation

Final results need to be reconciled with the authoritative source.

Thus, CDC allows faster propagation of changes but does not automatically guarantee the correctness of corporate metrics.

Apache Kafka 2.4.1: Support for Streaming Integrations

Apache Kafka 2.4.1 was released on March 12, 2020.

This is a corrective release of the 2.4 branch.

The newer Kafka 2.5 is not yet released: its release will take place on April 15, 2020.

For corporate analytics, Kafka can transmit relevant events from ERP, CRM, and other systems.

However, during business process changes, it is necessary to control not only technical delivery.

Event Contract

The message should have a clear structure and semantics.

Message Key

For related events, the key must be defined correctly.

Consumer Lag

Helps estimate lag.

Retention

Defines history availability.

Idempotency

Ensures safe re-application.

Data Quality

Controls the correctness of messages.

For example, if the ERP starts sending more canceled orders, Kafka can correctly deliver all events.

But DWH must correctly interpret their business meaning.

Therefore, timeliness should not precede the correctness of the semantic model.

Apache NiFi 1.11.4: Stability of Integration Processes

On March 22, 2020, Apache NiFi 1.11.4 was released.

This is a corrective release that contains fixes for issues with Parameter Contexts, SFTP processors, Site-to-Site resources, and other relevant components.

For businesses, these changes can be beneficial during the stabilization of information flows.

For example, under remote support conditions, it is important to have predictable integration configurations.

Parameter Contexts

Helps manage defined flow parameters.

SFTP

Used for relevant file integrations.

Site-to-Site

Can be used for transfer between NiFi environments.

Backpressure

Helps control queue filling.

Provenance

Supports FlowFile lifecycle analysis.

However, it should be noted that NiFi does not automatically determine the business correctness of received records.

This requires appropriate checks and approved information contracts.

Data Freshness: when the schedule no longer matches the business

One of the consequences of changing business processes can be the disruption of the usual information calendar.

For example, the ERP previously performed the relevant batch export at 01:00.

Now, part of the processes has changed, and the export becomes available later.

If the ETL continues to run at 01:15, it may receive an incomplete set.

Therefore, it is necessary to control not only the startup time but also the actual readiness of the source.

Source Ready Status

Has the corresponding operation in ERP been completed?

Source Watermark

What is the control limit for received changes?

Batch Completeness

Have all necessary data been received?

Publication Time

When did the result become available?

Business Calendar

Does the execution comply with the new schedule?

For example, an enterprise might abandon a rigid clock-based ETL start and use appropriate control conditions.

However, this requires proper orchestration.

It is especially important to correctly define the behavior if the source is not ready at the specified time.

For example, create an alert, do not publish an incomplete dataset, or apply an appropriate fallback process.

Data Reliability: How to Avoid Accumulating Information Problems

During rapid changes, an enterprise may create new information processes.

If they are implemented without proper control, technical debt gradually increases.

For example, each department creates its own copy of the client directory.

After some time, the information begins to diverge.

Another example: new SQL reports are generated without approved business methodologies.

Therefore, a Data Reliability model is needed.

Availability

Availability of necessary information systems.

Freshness

Data relevance.

Completeness

Completeness of relevant datasets.

Correctness

Correctness of business results.

Recoverability

Recoverability.

Traceability

Clear data lineage. It is important to evaluate these characteristics independently.

For example, SQL Server may be available, but the sales table will contain outdated records.

Therefore, infrastructure monitoring should be supplemented with information metrics.

Changing predictive models: why historical data can be misleading

For enterprises using Machine Learning, a sharp change in business conditions can create additional risks. For example, a company forecasts demand based on data from the previous three years.

The model accounts for seasonality, prices, and historical sales. However, current conditions can differ significantly.

In such a situation, the representativeness of certain historical patterns decreases.

Data Distribution Shift

The distribution of actual characteristics changes.

Concept Drift

The relationship between characteristics and the predicted outcome can change.

Model Monitoring

It is necessary to monitor the actual quality of forecasts.

Business Review

An assessment is needed to determine if the model’s assumptions remain relevant.

Retraining

Under appropriate conditions, the model may require retraining. However, automatic retraining on a very short, non-standard period can also create risks.

For example, a temporary drop in demand does not necessarily mean that all long-term patterns have ceased to apply. Therefore, an appropriate statistical and business assessment is needed.

Scenario Analytics Instead of a Single Forecast

In conditions of increased uncertainty, it may be beneficial for the enterprise to use several scenarios.

• For example:
• Baseline scenario. Operations continue with defined limitations.
• Adverse scenario. Part of the supplies is significantly delayed.
• Recovery scenario. Operational processes are gradually returning to a more familiar mode.

These scenarios are not predictions of future events. They are used to assess possible consequences.

For Data Warehouse, this creates the task of correctly distinguishing between actual and forecast data.

Actuals

Confirmed business results.

Forecasts

Forecast values according to relevant models.

Scenarios

Alternative sets of assumptions.

Versions

It is important to keep track of the forecast version and its creation date. For example, a forecast created at the beginning of March can differ significantly from a forecast at the end of the month.

This does not necessarily mean one of them is an error. It is possible that the information available at the time of calculation has changed.

Therefore, reproducibility is important for corporate analytics.

Data Quality Gates in Non-Standard Conditions

One of the key challenges is determining when an information set can be considered ready. For example, the DWH has received data from three ERPs, but the fourth is temporarily unavailable.

Should all reporting be blocked? The answer depends on the business purpose.

For official financial consolidation, an incomplete set may be unacceptable. However, for an operational dashboard, a partial result can be shown under certain conditions with clear indication of limitations.

Therefore, it is advisable to use multiple statuses.

This is a proposed corporate model.

It is not necessarily a built-in function of all DWHs.

Important Principle

Critical Data Quality checks should not be weakened just because business conditions have become more complex. Instead, the publication procedure can be changed, clearly distinguishing between preliminary and confirmed metrics.

This allows providing users with useful information without losing transparency.

Practical Scenario: Adapting a Trading Holding’s DWH

Consider a hypothetical holding company with several ERPs, a central CRM, MDM, and a Data Warehouse.

In March, the company changes some of its business processes.

Stage 1. Inventory of Changes

New statuses, channels, and operational rules are defined.

Phase 2. Business Glossary

Relevant definitions are updated.

Phase 3. Data Ownership

A responsible person is assigned for each change.

Phase 4. Data Contracts

Message structures and semantics are updated.

Phase 5. Impact Analysis

Impact on ETL, DWH, and BI is determined.

Phase 6. MDM

New corporate compliance checks are performed.

Stage 7. Data Profiling

Actual distribution changes are analyzed.

Stage 8. Regression Testing

Old and new business scenarios are checked.

Stage 9. Airflow

Relevant DAGs and control conditions are updated.

Stage 10. CDC / Kafka

Relevant contracts are checked for operational sources.

Stage 11. Data Quality Gates

Publication criteria are defined.

Stage 12. Operational Reports

Preliminary indicators are created as needed.

Stage 13. Certified DWH

Regulated datasets undergo necessary reconciliations.

Stage 14. Monitoring

Data Freshness and correctness are monitored.

Stage 15. Incident Management

The procedure for responding to new violations is defined.

Stage 16. Review

The team evaluates the practical result and improves the rules.

As a result, the company gains the ability to adapt the analytical architecture without losing manageability.

Organizational Responsibility During Changes

For Data Reliability, it is especially important to correctly delineate roles.

• The Data Owner approves business definitions.
• The Data Steward analyzes information exceptions.
• The Data Architect determines the impact on information models.
• The Data Engineer implements necessary transformations.
• The DBA ensures the reliability of the DBMS.
• The Integration Architect monitors relevant intersystem contracts.
• The BI Developer updates analytical representations.
• The Incident Coordinator organizes the response to critical incidents.

In non-standard conditions, these roles must have a clear remote interaction procedure. For example, a business owner can approve a new methodology without being physically present in the office, using the appropriate corporate process.

However, the decision must remain documented. Therefore, BPM and electronic document management can play an important role.

KPI Data Quality and Reliability during Changes

It is important to use KPIs in accordance with the business context. For example, a sharp drop in sales should not automatically be considered a Data Quality violation.

However, the absence of a mandatory ERP in the financial load can be a critical information problem. Therefore, automation should combine statistical analysis and approved business rules.

GDPR and Changes in Information Processes

For enterprises covered by GDPR, new information flows must take into account the relevant personal data processing rules. For example, a company is creating a new customer interaction channel.

Additional characteristics appear in the CRM. It is necessary to define their purpose and appropriate usage rules.

Data Minimisation

Additional personal attributes should not be collected unnecessarily.

Purpose Limitation

Processing must align with the defined purpose.

Data Quality

Inaccurate personal data must be processed according to relevant requirements.

Retention

Retention periods are defined for new datasets.

Access Control

Relevant permissions must be reviewed.

Data Lineage

Can help establish derivative resources. It is important not to assume that the speed of business changes allows for the abandonment of proper security procedures.

On the contrary, new information processes require appropriate analysis even at the design stage.

Intecracy Group Competencies in Data Reliability

Adapting corporate information architecture requires several technological competencies.

Data Management IG (dmig.com.ua) specializes in ETL, MDM, Data Quality, DWH, and corporate analytics. These areas are directly related to the analysis of information changes, automation of checks, reconciliation of directories, and support for the correctness of analytical indicators.

IQusion IT (iqusion.com) operates in the fields of system integration, electronic document management, and BPM. These competencies can be used to adapt business processes, approve new information rules, and organize relevant inter-system contracts.

Softline IT (softline.kiev.ua) has competencies in server, network infrastructure, and storage systems. They provide the foundation for reliable execution of ETL, DWH, Kafka, and other relevant components during load changes.

SL Global Service (sgs4business.com) specializes in cloud integration, virtualization, and hybrid infrastructure. These areas can be used for flexible scaling of relevant information resources and ensuring the availability of corporate analytics.

At the same time, decisions regarding changes in business methodologies, data criticality, and acceptable deviations must be approved by the enterprise.

Practical Implementation Guidelines

March 2020 demonstrates that corporate information architecture must be capable of adapting to rapid changes in business processes. Traditional Data Quality checks remain necessary, but they are insufficient to identify all information risks.

It is necessary to monitor Schema Drift, Semantic Drift, distribution changes, and the compliance of business contracts with new conditions.

Apache Airflow 1.10.9 can support controlled orchestration.

Apache Kafka 2.4.1 and Debezium 1.1 allow for the organization of relevant incremental and streaming integrations.

Apache NiFi 1.11.4 enhances the stability of enterprise information flows. However, technological speed should not replace Data Governance.

Critical metrics require approved methodologies, MDM, Reconciliation, and appropriate Data Quality Gates.

It is particularly important to distinguish between real business changes and technical disruptions.

The enterprise must be able to quickly update information rules without losing historical data and reliability.

The main principle of Data Reliability during sharp business changes is that corporate analytics should be able to not only correctly process data according to current rules but also to timely detect when these rules need revision and implement changes in a controlled manner – without distorting history, financial indicators, and business decisions.