Data Storage and Protection for Containerized Databases

Database Administration / Docker / Data Persistence

Why Database Containerization Requires a Special Approach

The widespread adoption of Docker is gradually changing how server applications are deployed. Developers gain the ability to create portable software images, quickly launch test environments, and ensure more predictable software configurations. However, when using containerization for enterprise databases, questions arise that are not always relevant for ordinary web services.

An application container can often be deleted and recreated from the same image. If it does not store critical state, such an operation, with proper configuration, does not lead to the loss of business information. For DBMS, the situation is fundamentally different.

PostgreSQL, MySQL, Oracle Database, and Microsoft SQL Server are used to store constantly changing information. The loss of even a small portion of confirmed transactions can have financial or operational consequences. Therefore, simply moving a DBMS into a container does not automatically ensure proper fault tolerance.

Database containerization only makes sense when an enterprise clearly separates the lifecycle of the software environment from the lifecycle of the corporate information.

Docker can simplify DBMS installation and updates, but data storage, backup, recovery, and replication policies must be designed separately.

Container and Database Have Different Lifecycles

A Docker Image contains the software environment required to run a corresponding container.

For example, a PostgreSQL image might include the DBMS server, standard utilities, and startup configurations.

A container is created from an image and gets its own writable filesystem layer.

During operation, the DBMS modifies tables, indexes, transaction logs, and service files.

If all this information is located only in the container’s writable layer, deleting the container will result in the loss of that layer.

Stopping is Not Deleting

It is important to distinguish between several operations.

Stopping a container using docker stop does not delete its writable layer by itself.

After restarting the same container, the corresponding files may remain accessible.

However, removing a container using docker rm results in the deletion of its own writable layer.

This is why storing a corporate database exclusively in this layer is dangerous.

Separating Data from the Image

For production use, it is necessary to ensure that the DBMS files remain accessible after the container is replaced. Docker offers several mechanisms for such storage.

The most important ones are Docker Volumes and bind mounts.

They allow data to be placed outside the container’s own writable layer. However, these mechanisms have different properties and administrative requirements.

Docker Volumes: Storing Data Independently of the Container

A Docker Volume is a separate storage resource that can be mounted to a container.

This allows the software environment to be replaced while preserving the data volume.

For example, a company can create a volume for PostgreSQL:

docker volume create pgdata

Then, start a container:

docker run -d \
  --name corporate-postgres \
  -e POSTGRES_PASSWORD=example_password \
  -v pgdata:/var/lib/postgresql/data \
  postgres:9.5

This is a conditional example for a test environment.

PostgreSQL 9.5 became generally available in January 2016, so the corresponding version can already be used in May. For a production system, the exact image release, security configuration, and backup mechanisms need to be additionally defined.

In the example, the pgdata volume contains the DBMS working files.

If the container is deleted, the named volume itself will be preserved separately during a normal docker rm operation.

A new container can use the same volume, provided that the versions are compatible and the configuration is correct.

However, it is necessary to consider that the volume is not a backup.

If the information within it is accidentally modified or corrupted, Docker will not automatically restore the database to its previous state.

Bind Mounts: Direct Use of Host File System

Another mechanism is bind mount.

It allows connecting a defined directory of the host operating system inside the container.

For example:

docker run -d \
  --name corporate-postgres \
  -e POSTGRES_PASSWORD=example_password \
  -v /srv/postgresql/data:/var/lib/postgresql/data \
  postgres:9.5

In this example, PostgreSQL working files are stored in the /srv/postgresql/data directory on the host.

The advantage is that the administrator can directly define the data location.

This can be useful if the enterprise has an established disk resource structure or uses a specialized storage system.

However, bind mounts require attention to access rights, file ownership, and operating system configuration.

File Resource Permissions

The DBMS process inside the container runs with a specific user ID.

If the host directory owner or file system permissions do not meet the process requirements, the database may fail to start or operate incorrectly.

It is especially important to check these issues when migrating containers between servers.

Host Dependency

Bind mount uses a specific file system path.

If the container is migrated to another server, it is necessary to ensure the presence of the corresponding directory and data.

Thus, the portability of a Docker image does not imply the automatic portability of the entire information system.

Docker and Enterprise DBMS Performance

Disk subsystem performance is of great importance for databases.

Transactional DBMS perform operations such as log writes, page updates, checkpoint creation, and synchronization of information with storage devices. Therefore, it is necessary to evaluate I/O latencies and guarantees for the persistence of written data.

Docker File Drivers

Docker uses storage drivers to work with image layers and the container’s writable filesystem.

In Linux environments, aufs, devicemapper, overlay, and other supported drivers may be used depending on the configuration.

However, it is advisable to place the DBMS’s working data on a specially prepared volume or filesystem, rather than relying on the container’s writable layer. This allows separating the database’s intensive disk load from the container’s state change mechanisms.

Real Load Testing

It cannot be stated that Docker necessarily provides the same performance as installing the DBMS directly on the server.

The result depends on the driver, filesystem, kernel configuration, disk subsystem, and the nature of the operations. Therefore, testing is necessary.

For PostgreSQL, pgbench can be used, and for MySQL, corresponding SQL load generation tools, such as sysbench.

During testing, it’s important to replicate real-world scenarios: transaction intensity, number of concurrent users, data volume, and the nature of SQL queries.

A test with a small, empty database doesn’t always reflect the behavior of a production system.

Backing Up Containerized Databases

The most dangerous mistake is to try to consider a Docker image copy as a database backup.

An image contains the software environment but doesn’t necessarily include the current data stored in attached volumes. Similarly, simply copying files from a running DBMS doesn’t always guarantee a consistent result.

Therefore, backups must be performed considering the mechanisms of the specific database management system.

Logical Backup of PostgreSQL

For PostgreSQL 9.5, you can use pg_dump.

For example, to create a logical copy of a specific database:

docker exec corporate-postgres \
  pg_dump -U postgres -Fc corporate_db \
  > corporate_db.backup

In this example, the command is executed from the host, and the result is written to a file in the current host directory.

The -Fc format creates an archive suitable for restoration using pg_restore.

This approach is convenient for individual databases, test environments, and defined migration scenarios. However, for large production systems, it may be insufficient.

Logical backup does not replace physical backup mechanisms and point-in-time recovery.

Physical Backup and WAL

PostgreSQL supports physical base backups and Write-Ahead Log archiving.

The combination of these mechanisms allows for information recovery up to a defined point in time, provided the necessary log chain is available.

In a containerized environment, it is necessary to ensure reliable storage not only of the main DBMS files but also of archived WAL (Write-Ahead Logging). It is especially important that backups are stored independently of the Docker host.

If the server with containers fails completely, the local volume may become inaccessible.

MySQL

For MySQL, mysqldump, binary logs, and appropriate physical backup tools can be used.

The specific choice depends on the MySQL version, table types, and requirements for acceptable data loss. It is also important to verify that the backup contains all necessary objects: table structures, data, procedures, and other components required for application recovery.

Volumes, Backups, and Disaster Recovery — Different Mechanisms

For enterprise operation, it is necessary to clearly distinguish between these three tasks.

For example, if a user mistakenly deletes a critical PostgreSQL table, the operation can be replayed on a backup replica.

In such a case, replication will not replace a backup.

Similarly, if a container operates with a local volume, automatically starting a new container on another server does not guarantee access to the same data. An appropriate storage or recovery system must be provided.

PostgreSQL and MySQL Replication in a Containerized Environment

Containerization does not negate the replication mechanisms available in DBMS.

PostgreSQL 9.5 supports streaming replication based on WAL.

MySQL 5.6 supports Binary Log-based replication, including the ability to use Global Transaction Identifiers.

For containerized deployment, it is necessary to define node network addresses, authentication rules, and independent data storage.

Primary and Standby

• The primary PostgreSQL server accepts transactional operations.
• The standby node receives logs and replays changes.
• Containers can be used as execution environments for corresponding processes.

However, Docker itself does not determine which node should be primary after a failure. This requires separate procedures or specialized management tools.

Split-brain Danger

If two nodes simultaneously start accepting independent changes, a conflicting information state can arise. This is especially dangerous for financial and transactional systems.

Therefore, automatic container restart cannot be equated to safe DBMS failover.

It is necessary to ensure role control, primary node availability checks, and mechanisms to prevent concurrent writes.

Orchestration and Container Environment Limitations

Docker Swarm and Kubernetes are available for managing containers across multiple servers.

In May 2016, Docker Swarm 1.2 already supported container rescheduling mechanisms in case of node failures. Kubernetes 1.2, released in March, also enhances cluster management capabilities.

However, for databases, it is necessary to consider state storage specifics. If the DBMS container restarts on another server, its local volume is not necessarily available.

This means that the orchestrator can resume the process, but does not guarantee information recovery.

Local and Network Storage

For a clustered environment, centralized storage systems or specialized network resources can be used. However, their operation creates additional requirements for performance, availability, and record consistency.

For example, if a database uses network storage, latency can affect transaction commit speed. In addition, not every network file system is suitable for a specific DBMS.

Compatibility must be checked according to the manufacturer’s documentation and practical tests.

Automatic Restart

Docker supports container restart policies. This can be useful for resuming a process after it completes with an error.

However, restarting does not always eliminate the cause of failure. If the DBMS cannot access the disk or its files are corrupted, repeated startup attempts may only create additional load.

Therefore, it is necessary to combine automation with monitoring and diagnostics.

Information Security of Containerized DBMS

A corporate database often contains financial information, personal data, and other critical records.

Therefore, the security of the container environment must be considered at several levels.

Network Access

The DBMS port should not be unnecessarily published on all host network interfaces. For example, if PostgreSQL is only used by an internal application service, access to it can be restricted to the appropriate Docker network.

Additionally, it is necessary to use the DBMS’s native authentication and the corporate infrastructure’s network rules.

Container Privileges

For databases, --privileged should not be used without a valid reason. This parameter significantly expands the container’s capabilities for accessing host resources.

It is also necessary to correctly configure file system permissions and system users.

Docker Engine 1.11 continues the development of user namespaces and seccomp mechanisms, which can help limit process privileges.

However, the configuration of these mechanisms must be compatible with the requirements of the specific DBMS.

Credentials

Database user passwords should not be written directly into the Dockerfile. It is also undesirable to store them in open configuration files accessible to a wide range of employees.

In May 2016, one should not rely on built-in Docker Secrets mechanisms related to later platform capabilities.

Therefore, it is necessary to use existing corporate tools for protecting configuration, file permissions, and credential management.

Updating Database Versions in Containers

One of the advantages of Docker is the ability to create a new container with the required software version. However, for DBMS, updates are not always reduced to replacing the image.

For example, PostgreSQL 9.4 data cannot be unconditionally connected to a PostgreSQL 9.5 container and expected to work correctly. Internal data formats and system catalogs can change between major PostgreSQL versions.

For such a transition, a supported upgrade procedure must be used, such as pg_upgrade or logical replication.

Upgrade Plan

For a corporate database, a sequence of actions must be defined: creating a backup, checking compatibility, performing a test upgrade, estimating downtime, and preparing a rollback plan.

Drivers, extensions, and application queries also need to be checked.

Rollback

Returning to a previous Docker image does not always ensure the DBMS is restored to its previous state. If the new version has changed the internal database structure or the application has performed table migrations, the previous container may become incompatible with the data.

Therefore, for critical upgrades, an independent backup must be created, and the rollback procedure must be verified.

Practical Scenario: Docker for Development and Testing Environments

Let’s consider a hypothetical enterprise developing a corporate document management service. The production PostgreSQL database is hosted on a dedicated server.

The development team has several test environments, each requiring the same DBMS version. Previously, the administrator manually installed PostgreSQL and configured parameters on each server.

Standardization

The team creates a defined PostgreSQL 9.5 Docker image with the necessary parameters for testing.

Developers can run the same environment on their own Linux servers or in corresponding virtual machines.

Test Data Preparation

Specially prepared datasets are used for testing. If it is necessary to reproduce a production issue, the data undergoes appropriate cleaning or anonymization.

This allows for the non-distribution of full copies of confidential corporate databases between workstations.

Automation

Jenkins launches the DBMS container, performs the necessary structure migrations, and runs integration tests. After the checks are completed, the test environment can be removed.

This is acceptable for temporary test data if process rules do not require its preservation.

Production Environment

The main corporate DBMS continues to operate on specialized infrastructure. The enterprise gains the benefits of containerization in the development process without simultaneously changing the production storage architecture.

This approach can be a rational first step in using Docker.

When it is advisable to containerize a production database

The decision depends on the scale and nature of the information system.

For an auxiliary web application with a small database, containerization can simplify deployment and maintenance. For a critical ERP with a large transactional load, the benefits of this approach must be assessed separately.

Performance, backup features, support for the DBMS used, and disaster recovery capabilities must be considered. It is especially important to check the official requirements of the software vendor.

For example, containerizing a proprietary DBMS may have licensing and technical limitations.

It is also necessary to distinguish between experimental images and configurations for which the vendor guarantees a corresponding level of support.

Alternative: Containerized Applications and External DBMS

For many enterprises, an architecture where APIs, web services, and auxiliary application components are containerized, while the central database remains in its usual environment, is advisable. Containers connect to the DBMS via secure network interfaces.

This allows for independent updates of application services and the preservation of existing corporate data administration procedures.

Organizational Requirements and Data Governance

Containerization does not change the business responsibility for information. The Data Owner defines the rules for using a specific category of corporate data.

The Data Steward controls their quality. The DBMS Administrator is responsible for technical operation, backup, recovery, and performance.

The Infrastructure team ensures the operation of Docker hosts, networks, and storage systems. It is important to clearly define the boundaries of responsibility.

For example, if a PostgreSQL container loses access to a volume, the administrator must have a clear procedure for interacting with those responsible for the storage system. It is also necessary to document which containers use specific information resources.

This allows for assessing the impact of infrastructure changes and avoiding accidental deletion of critical volumes.

Intecracy Group Competencies in Containerized Data Operations

Deploying corporate databases in containers requires a combination of several technological specializations. Data Management IG focuses on corporate data management, ETL, MDM, Data Quality, DWH, and analytics. These competencies are crucial for ensuring the reliability, consistency, and controlled migration of information between environments.

IQusion IT specializes in system integration, electronic document management, and BPM. These competencies are relevant for creating containerized application services that utilize corporate DBMS. Softline IT possesses expertise in server and network infrastructure, storage systems, and corporate infrastructure projects. This is particularly important when selecting disk architecture, backup strategies, and ensuring database performance.

SL Global Service specializes in cloud integration, virtualization, and hybrid infrastructure. Such competencies can be applied when deploying containerized environments across various computing platforms.

At the same time, the architectural decision should be guided by data criticality and the enterprise’s recovery requirements.

Practical Implementation Guidelines

Docker enables the standardization of deployment not only for application services but also for individual databases. However, using containers does not eliminate the fundamental requirements for corporate DBMS.

For reliable operation, it is necessary to separate the database working files from the container’s writable layer, use appropriate volumes, and organize independent backups. Performance monitoring of the disk subsystem, access control, secure version updates, and disaster recovery testing are of particular importance.

Containerization can be particularly useful for development, testing, and auxiliary information system environments.

For large transactional platforms, its feasibility must be confirmed by practical trials and risk assessment.

The container should remain a replaceable software environment, while corporate data must be stored, backed up, and restored independently of its lifecycle.