Zero-Downtime Linux Clustering: Building an Active-Passive MySQL Cluster with Corosync and Pacemaker

Key Architecture Takeaways

• ›
  Eliminate Split-Brain with Dual-Ring Corosync: Deploying dual redundant Totem communication rings across independent physical network interfaces guarantees that transient link degradation cannot trigger dual-primary state divergence.
• ›
  Shared-Nothing Synchronous Storage (DRBD 9 Protocol C): Replicating database block devices over a dedicated point-to-point link with synchronous write barriers ensures 0-byte transaction data loss across primary node crashes.
• ›
  Strict Node Fencing via STONITH: Operating an active-passive stateful database cluster without Shoot The Other Node In The Head (STONITH) guarantees data corruption; Pacemaker must physically reset partitioned nodes via Proxmox PVE API or IPMI BMC before promoting secondary storage.
• ›
  Colocation & Ordered Resource Orchestration: Pacemaker resource sets enforce atomic dependency bindings: the Virtual IP (10.10.10.100), DRBD primary promotion, /var/lib/mysql filesystem mount, and mysqld service must always bind to the same voting node in strict sequential order.

High-Availability Active-Passive Failover Architecture

In production database environments running bare-metal servers or private KVM/Proxmox hypervisors, single-node hardware failures—such as power supply blowouts, kernel panics, or memory bus errors—can lead to catastrophic unplanned downtime.

While active-active multi-master systems (like MySQL Galera or NDB Cluster) introduce complex write latency penalties and distributed locking overhead, an Active-Passive High-Availability Cluster using Corosync, Pacemaker, and DRBD 9 delivers maximum single-thread write throughput with instantaneous, automated failover.

Agentic Infrastructure Automation & Verification

Engineering enterprise clustering configurations and database client connectors requires flawless precision. A single misplaced constraint or unhandled socket disconnect during a 10-second STONITH failover window can exhaust client connection pools and cause total service outages.

At TechNHIT, clustering blueprints, client failover connectors, and multi-node validation drills are designed, scaffolded, and continuously validated using the Google Antigravity MDX Agentic Workflow Engine. Below is the complete interactive infographic detailing how declarative recipe files translate into automated, verified infrastructure artifacts:

<Callout type="tip" title="Interactive Agentic Workflow Simulator Available"> You can now experience this entire multi-agent pipeline live in your browser. Switch to Simulation Mode using the top tab bar or launch the console to watch the Planner, Coder, and Evaluator agents run tests, trigger self-healing loops upon cluster failover regressions, and pause at human approval gates.

Launch the Interactive Simulation Console (?mode=simulation) </Callout>

Declarative Recipe Specification: Provision Active-Passive MySQL Cluster

Below is the complete, production-grade MDX workflow file (provision-active-passive-mysql.mdx) governing the automated setup, storage synchronization, and catastrophic failover validation for this two-node HA Linux database cluster:

---
name: "Provision Active-Passive MySQL Cluster (Corosync + Pacemaker + DRBD)"
description: "End-to-end automated setup, configuration, and failover validation for a two-node HA Linux database cluster"
tools:
  - "mcp:terminal"
  - "mcp:fs"
  - "mcp:network"
parameters:
  node_primary_ip:
    type: string
    description: "Private heartbeat IP of Node A (e.g., 10.10.10.11)"
    required: true
  node_secondary_ip:
    type: string
    description: "Private heartbeat IP of Node B (e.g., 10.10.10.12)"
    required: true
  virtual_ip:
    type: string
    description: "Cluster Virtual IP / Floating IP (e.g., 10.10.10.100)"
    required: true
  drbd_device:
    type: string
    default: "/dev/drbd0"
  drbd_disk_partition:
    type: string
    default: "/dev/sdb1"
  fencing_agent:
    type: string
    description: "Fencing mechanism (e.g., fence_ipmilan, fence_pve, fence_virsh)"
    default: "fence_ipmilan"
---

# High-Availability MySQL Cluster Orchestration

<Context>
You are an autonomous Site Reliability Engineer and Linux Systems Architect inside Google Antigravity.
Your objective is to provision, configure, and validate a resilient, zero-downtime active-passive MySQL database cluster across two Linux nodes using Corosync for cluster messaging, Pacemaker as the CRM, DRBD for block-level replication, and STONITH for split-brain prevention.

Target Virtual IP: {{ parameters.virtual_ip }}
Heartbeat Nodes: {{ parameters.node_primary_ip }} (Node A), {{ parameters.node_secondary_ip }} (Node B)
Storage Device: {{ parameters.drbd_device }} on {{ parameters.drbd_disk_partition }}
Fencing Agent: {{ parameters.fencing_agent }}
</Context>

---

## Step 1: Pre-Flight Discovery & Network Verification

<Task id="recon_and_prerequisites" agent="infra_planner">
1. Audit local network interfaces across both nodes:
   - Verify low-latency dedicated heartbeat interfaces (Ring 0 on 10.10.10.x and Ring 1 on 10.20.20.x).
   - Confirm hostname resolution via /etc/hosts on both nodes:
     ```bash
     grep -E 'db-node-01|db-node-02' /etc/hosts
     ```
   - Ensure TCP ports 7788 (DRBD), 2224 (pcsd), and UDP ports 5404, 5405 (Corosync) are open on firewall/ufw/iptables.
2. Check shared-nothing storage block device availability:
   - Run `lsblk` and verify {{ parameters.drbd_disk_partition }} exists unmounted on both nodes.
3. Validate NTP/Chrony synchronization between nodes (clock skew must be < 10ms).
</Task>

---

## Step 2: DRBD Synchronous Storage Provisioning

<Task id="configure_drbd" agent="storage_engineer" dependsOn="recon_and_prerequisites">
1. Generate the DRBD configuration file at `/etc/drbd.d/mysql-ha.res`:
   - Protocol: C (Synchronous zero-loss replication).
   - Configure split-brain handlers (discard-node-nodata, adjust-my-split-brain).
   - Define endpoints for Node A ({{ parameters.node_primary_ip }}) and Node B ({{ parameters.node_secondary_ip }}).
2. Initialize metadata and attach storage on both nodes:
   ```bash
   drbdadm create-md mysql-ha
   drbdadm up mysql-ha
   ```
3. Establish initial sync on Node A:
   ```bash
   drbdadm primary --force mysql-ha
   ```
4. Format the filesystem on {{ parameters.drbd_device }} with ext4 (Multi-Mount Protection enabled).
5. Gate: Do not proceed until `dstate:UpToDate/UpToDate` is achieved.
</Task>

---

## Step 3: Corosync, Pacemaker & STONITH Initialization

<Task id="corosync_pacemaker_stonith" agent="cluster_architect" dependsOn="configure_drbd">
1. Authenticate nodes and initialize cluster:
   ```bash
   pcs host auth db-node-01 db-node-02 -u hacluster
   pcs cluster setup mysql_cluster db-node-01 addr=10.10.10.11 addr=10.20.20.11 db-node-02 addr=10.10.10.12 addr=10.20.20.12
   pcs cluster start --all && pcs cluster enable --all
   ```
2. Configure STONITH fencing devices (IPMI BMC or Proxmox VE API):
   ```bash
   pcs stonith create fence_node1 {{ parameters.fencing_agent }} ip=192.168.10.51 pcmk_host_list=db-node-01
   pcs stonith create fence_node2 {{ parameters.fencing_agent }} ip=192.168.10.52 pcmk_host_list=db-node-02
   ```
3. Enforce STONITH enabled (`stonith-enabled=true`) and no-quorum-policy=stop.
</Task>

---

## Step 4: Cluster Resource Configuration & Constraints

<ApprovalGate id="cluster-constraint-review" dependsOn="corosync_pacemaker_stonith">
  <Condition>
    Corosync quorum formed and STONITH active. Operator must verify colocation and ordering rules.
  </Condition>
  <Prompt>
    Verify that MySQL and Filesystem colocate with DRBD Master and follow strict promotion order.
  </Prompt>
</ApprovalGate>

<Task id="activate_resources" agent="crm_operator" dependsOn="cluster-constraint-review">
1. Define cluster resources:
   - VIP (ocf:heartbeat:IPaddr2, ip=10.10.10.100)
   - fs_mysql (ocf:heartbeat:Filesystem, device=/dev/drbd0, directory=/var/lib/mysql)
   - mysql_svc (ocf:heartbeat:mysql, config=/etc/mysql/my.cnf)
2. Enforce atomic colocation & ordering:
   ```bash
   pcs constraint colocation add VIP with fs_mysql INFINITY
   pcs constraint colocation add fs_mysql with ms_drbd_mysql:Master INFINITY
   pcs constraint order promote ms_drbd_mysql then start fs_mysql then start VIP
   ```
</Task>

---

## Step 5: Automated Failover & Self-Healing Testing

<Task id="failover_validation" agent="reliability_evaluator" dependsOn="activate_resources">
1. Inject simulated catastrophic kernel panic on Node A:
   ```bash
   ssh db-node-01 'echo c > /proc/sysrq-trigger'
   ```
2. Validate automated failover execution:
   - Corosync detects token loss in < 1000ms.
   - Node B invokes STONITH via IPMI: Node A power cut attested.
   - Node B promotes DRBD /dev/drbd0 to Primary.
   - Node B mounts /var/lib/mysql and starts MySQL.
   - Node B takes over Virtual IP (10.10.10.100) and broadcasts Gratuitous ARP.
3. Validate total recovery time objective: RTO < 10s, RPO = 0 transactions.
</Task>

The Three Core Pillars of Active-Passive Linux Clustering

1. Heartbeat Redundancy & Split-Brain Prevention (Corosync 3)

Corosync is the cluster messaging and membership layer. It maintains the Totem Single Ring / Multiple Ring Redundant Protocol, passing heartbeat tokens continuously between cluster members.

If a network switch fails and the nodes cannot see each other, each node might assume the other is dead and attempt to promote its local storage. This is known as Split-Brain Syndrome. If both nodes become primary and write to their local copies of the database simultaneously, the filesystem and InnoDB tablespaces suffer unrecoverable divergence.

To prevent split-brain: 1. Redundant Dual-Ring Topology: Corosync 3 uses Kronosnet (knet) to transport heartbeat tokens across two completely separate physical networks simultaneously: - Ring 0 (Management / Ingress): 10.10.10.0/24 via top-of-rack switches. - Ring 1 (Private Heartbeat / Storage Crossover): 10.20.20.0/24 via direct point-to-point 10GbE DAC/SFP+ cabling. 2. Quorum Device (QDevice / QNetd): In a 2-node cluster, 1 node failure leaves the remaining node with only 50% of the vote (no majority). A lightweight external witness daemon (corosync-qdevice on a separate Raspberry Pi or cloud VM) acts as the tiebreaker, providing the 3rd vote needed to form legal quorum.

2. Shared-Nothing Block Storage (DRBD 9 Protocol C)

Traditional clustering relied on costly shared SAN/NAS arrays (iSCSI, Fibre Channel), which themselves introduced a single storage controller point of failure. DRBD (Distributed Replicated Block Device) operates as a Linux kernel driver that mirrors block devices (/dev/drbd0) over high-speed networks, creating virtual shared storage with zero shared hardware.

DRBD operates in three distinct replication modes: - Protocol A (Asynchronous): Writes are acknowledged locally as soon as they hit local disk and the TCP send queue. Fast over high-latency WANs, but risks data loss on sudden power failure. - Protocol B (Memory Synchronous): Writes are acknowledged once received in peer RAM buffer. - Protocol C (Synchronous): Writes are only acknowledged to the application (MySQL) when written to both local persistent storage and peer remote storage. This is mandatory for high-availability database clusters.

3. Absolute Node Isolation (STONITH Fencing)

In high-availability clustering, fencing is the non-negotiable mechanism that isolates an unresponsive node.

If db-node-01 experiences a temporary 10-second kernel CPU lockup or high load spike, it might drop Corosync heartbeats. If db-node-02 blindly promoted itself while db-node-01 was still flushing dirty pages to the block device, catastrophic data destruction occurs.

STONITH (Shoot The Other Node In The Head) ensures that before db-node-02 can promote DRBD or bind the Virtual IP, it issues a hardware command (via IPMI/iLO/iDRAC BMC or Proxmox VE API fence_pve) to forcibly yank power from db-node-01 or issue an immediate hardware reboot. Only after receiving hardware attestation that db-node-01 is dead will Pacemaker complete the failover.

Step-by-Step Implementation Guide

<Steps> <Step title="Prerequisites and Network Interface Blueprint"> Configure the network interfaces on both nodes. Both nodes run Ubuntu 24.04 LTS or Debian 12 with two dedicated physical NICs.

<CodeTabs> <div label="/etc/netplan/01-netcfg.yaml (db-node-01)" filename="/etc/netplan/01-netcfg.yaml">

network:
  version: 2
  renderer: networkd
  ethernets:
    # Public / Management LAN (Ring 0)
    eth0:
      addresses: [10.10.10.11/24]
      routes:
        - to: default
          via: 10.10.10.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]
    # Dedicated Storage & Heartbeat Direct Crossover (Ring 1)
    eth1:
      addresses: [10.20.20.11/24]
      mtu: 9000

</div>

<div label="/etc/netplan/01-netcfg.yaml (db-node-02)" filename="/etc/netplan/01-netcfg.yaml">

network:
  version: 2
  renderer: networkd
  ethernets:
    # Public / Management LAN (Ring 0)
    eth0:
      addresses: [10.10.10.12/24]
      routes:
        - to: default
          via: 10.10.10.1
      nameservers:
        addresses: [1.1.1.1, 8.8.8.8]
    # Dedicated Storage & Heartbeat Direct Crossover (Ring 1)
    eth1:
      addresses: [10.20.20.12/24]
      mtu: 9000

</div> </CodeTabs>

Apply the configuration and verify low-latency ping across the direct link:

<Step title="Install Clustering and Replication Packages"> Install Pacemaker, Corosync, DRBD 9 kernel modules, and fencing agents on both nodes:

Stop the standalone MySQL daemon and disable its default systemd auto-start. Pacemaker must control MySQL lifecycle exclusively:

<Step title="Configure DRBD 9 Shared-Nothing Block Replication"> Ensure each node has an identical backing raw partition or LVM logical volume (e.g., /dev/vg_data/lv_mysql). Create the DRBD resource definition on both nodes:

<CodeTabs> <div label="/etc/drbd.d/r0.res" filename="/etc/drbd.d/r0.res">

resource r0 {
    protocol C;

    net {
        cram-hmac-alg sha256;
        shared-secret "TechNHIT-Cluster-Drbd-Secret-Key-2026";
        max-epoch-size 8000;
        max-buffers 8000;
        sndbuf-size 2M;
        rcvbuf-size 2M;
        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;
    }

    disk {
        on-io-error detach;
        fencing resource-and-stonith;
        al-extents 3389;
    }

    on db-node-01 {
        device /dev/drbd0;
        disk /dev/vg_data/lv_mysql;
        address 10.20.20.11:7788;
        meta-disk internal;
    }

    on db-node-02 {
        device /dev/drbd0;
        disk /dev/vg_data/lv_mysql;
        address 10.20.20.12:7788;
        meta-disk internal;
    }
}

</div> </CodeTabs>

Initialize the DRBD metadata on both nodes and bring up the device:

On db-node-01 ONLY, force the initial sync to declare it the primary source of truth:

Monitor the synchronization progress until both nodes report UpToDate/UpToDate:

Format the replicated device /dev/drbd0 with Ext4 and copy over existing MySQL database data on db-node-01:

Demote DRBD on db-node-01 so that Pacemaker can manage master/slave roles dynamically:

<Step title="Initialize Corosync 3 Dual-Ring Cluster via PCS"> Set the cluster administrator password (hacluster) on both nodes:

From db-node-01, authenticate and create the two-ring Corosync cluster:

Verify that both knet links (Ring 0 and Ring 1) are online and active:

<Step title="Configure STONITH Fencing Devices"> Configure the hardware fencing agents. For virtualized nodes on Proxmox VE 8, use fence_pve. For bare-metal servers, use fence_ipmilan.

<CodeTabs> <div label="Proxmox VE Fencing (fence_pve)" filename="pcs-stonith-pve.sh">

# Create fencing agent for db-node-01 (VMID 101 on Proxmox node pve-01)
sudo pcs stonith create fence_node01 fence_pve \
    ipaddr="10.10.10.2" \
    login="root@pam" \
    passwd="ProxmoxRootPassword!" \
    vmid="101" \
    pcmk_host_list="db-node-01" \
    op monitor interval=60s

# Create fencing agent for db-node-02 (VMID 102 on Proxmox node pve-02)
sudo pcs stonith create fence_node02 fence_pve \
    ipaddr="10.10.10.3" \
    login="root@pam" \
    passwd="ProxmoxRootPassword!" \
    vmid="102" \
    pcmk_host_list="db-node-02" \
    op monitor interval=60s

</div>

<div label="Bare-Metal IPMI Fencing (fence_ipmilan)" filename="pcs-stonith-ipmi.sh">

# IPMI BMC Fencing for Node 1
sudo pcs stonith create fence_node01 fence_ipmilan \
    ipaddr="192.168.1.101" \
    login="ADMIN" \
    passwd="BmcSecretPassword" \
    lanplus=1 \
    action="reboot" \
    pcmk_host_list="db-node-01" \
    op monitor interval=60s

# IPMI BMC Fencing for Node 2
sudo pcs stonith create fence_node02 fence_ipmilan \
    ipaddr="192.168.1.102" \
    login="ADMIN" \
    passwd="BmcSecretPassword" \
    lanplus=1 \
    action="reboot" \
    pcmk_host_list="db-node-02" \
    op monitor interval=60s

</div> </CodeTabs>

Verify that STONITH devices are active and monitoring:

<Step title="Orchestrate Pacemaker Cluster Resources & Constraints"> Now, register the four core resources into Pacemaker: 1. DRBD Data Resource (ocf:linbit:drbd with promotable clone). 2. Filesystem Resource (ocf:heartbeat:Filesystem mounting /dev/drbd0 to /var/lib/mysql). 3. Virtual IP Resource (ocf:heartbeat:IPaddr2 managing 10.10.10.100). 4. MySQL Service Resource (systemd:mysql).

Execute the complete orchestration block via pcs:

Enforce Colocation Constraints (all resources must run on the active node that holds DRBD Master):

Enforce Strict Startup/Shutdown Ordering Constraints (DRBD Promoted ➔ Filesystem Mounted ➔ VIP Bound ➔ MySQL Started):

<Step title="Validate Cluster Quorum and Live Status"> Verify that all resources have converged into their designated states on db-node-01:

Expected status output: ``` Cluster name: technhit_ha_db Cluster Summary: Stack: corosync Current DC: db-node-01 (version 2.1.6) - partition with quorum Last updated: Sun Oct 25 14:00:00 2026 Last change: Sun Oct 25 13:50:00 2026 by root via cibadmin on db-node-01 2 nodes configured 6 resource instances configured

Node List: * Online: [ db-node-01 db-node-02 ]

Full List of Resources: fence_node01 (stonith:fence_pve): Started db-node-02 fence_node02 (stonith:fence_pve): Started db-node-01 Clone Set: drbd_mysql-clone [drbd_mysql] (promotable): Masters: [ db-node-01 ] Slaves: [ db-node-02 ] fs_mysql (ocf::heartbeat:Filesystem): Started db-node-01 vip_mysql (ocf::heartbeat:IPaddr2): Started db-node-01 svc_mysql (systemd:mysql): Started db-node-01 ``` </Step> </Steps>

Simulated Catastrophic Hardware Failure & Disaster Recovery Drill

To attest that our clustering architecture withstands sudden real-world hardware failures without operator intervention, we execute a destructive failover drill:

Phase 1: Simulate Sudden Power Loss on db-node-01 Trigger an immediate kernel crash / hardware reset on the active node without allowing MySQL to gracefully unmount or flush buffers:

Phase 2: Autonomous Cluster Reaction Sequence (Timeline: 8.2 Seconds) 1. T+0.0s: Corosync token heartbeat drops on both Ring 0 and Ring 1. 2. T+1.5s: Pacemaker on db-node-02 notices missing heartbeats and queries QDevice witness for quorum consensus. 3. T+2.2s: Quorum confirmed. db-node-02 immediately triggers STONITH fence agent (fence_pve / fence_ipmilan), commanding the BMC/hypervisor to hard-kill db-node-01's power. 4. T+3.4s: STONITH confirms db-node-01 is severed from the storage fabric. 5. T+4.1s: db-node-02 executes drbdadm primary r0, promoting its local block device to Master. 6. T+5.3s: Pacemaker mounts /dev/drbd0 onto /var/lib/mysql. Journal recovery runs in milliseconds thanks to synchronous Protocol C writes. 7. T+6.5s: Virtual IP 10.10.10.100 binds to eth0:0 on db-node-02 and sends Gratuitous ARP packets to the network switch. 8. T+8.2s: MySQL systemd service starts. Application database connection pools resume servicing queries with zero data loss.

Phase 3: Node Re-Integration and Non-Destructive Rejoin When db-node-01 reboots, Corosync re-establishes knet heartbeats. DRBD automatically detects that db-node-02 advanced the generation UUID, attuning db-node-01 as Secondary and streaming incremental delta blocks to catch up.

Frequently Asked Questions (AI & Engineering FAQ)

Why is running a 2-node cluster without STONITH considered dangerous? Without STONITH fencing, any network glitch or switch reboot severs cluster communication while both physical nodes remain running. Both nodes assume the other is dead and attempt to mount the filesystem and write to MySQL simultaneously. Because DRBD cannot merge divergent block writes, this causes catastrophic split-brain corruption, permanently corrupting InnoDB transaction tables.

What is the difference between DRBD Protocol A, B, and C? - Protocol A (Async): Write completes locally before peer acknowledges. High throughput across WANs, but data loss occurs on crash. - Protocol B (Semi-Sync): Write is received in peer network buffers. - Protocol C (Sync): Write must be committed to physical disk storage on both nodes before returning success to MySQL. This provides 100% zero-transaction-loss durability for enterprise databases.

How does Pacemaker handle Virtual IP (VIP) migration? Pacemaker utilizes the IPaddr2 resource agent. When failover occurs, it adds the IP alias (10.10.10.100) to the active node's physical network adapter and broadcasts Gratuitous ARP (GARP) frames to neighboring top-of-rack switches, instantly updating switch MAC address lookup tables with zero router reconfiguration.

External Architectural Breakdowns & Suggested Reading