How to create a HA MediaWiki

      No Comments on How to create a HA MediaWiki

Couple of weeks ago we had to simulate a disaster in one of our datacenters. During the simulation we had to test several application for availability and I noticed that our MediaWiki wasn’t available. The reason why was simple it was only available in our primary datacenter. In this case it wasn’t a big deal cause is was only a simulation, but when there is a real disaster a MediaWiki is in our case very helpful.

So I was looking for a solution to make our MediaWiki High Available. Unfortunately I couldn’t find a proper one. So I decided to create my own HA MediaWiki with the help of DRBD replication between 2 servers and an external MySQL Cluster.

What is DRBD?
The Distributed Replicated Block Device (DRBD) is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between hosts.

MySQL Cluster
How to create a MySQL cluster? You can find it right here.
MySQL (Master-Master) cluster with High Availability – Part 1 of 2

Note: This guide is written for Debian 8 and written for a non-root user.
Commands that require elevated privileges are prefixed with [sudo]

Servers

WIKI01:
LAN-Eth0: 10.40.0.15/24
REPL-Eth1: 10.90.0.5/24
Additional 5GB Harddisk for DRDB

WIKI02:
LAN-Eth0: 10.40.0.16/24
REPL-Eth1: 10.90.0.6/24
Additional 5GB Harddisk for DRDB

Cluster-IP: 10.40.0.17/24

Install & Configure Apache and HDD on WIKI01 & WIKI02

First we need to install NTP, Apache and all requirements for MediaWiki

user@WIKI01:~$ sudo apt-get install ntp ntpdate apache2 \
php5 php5-mysql libapache2-mod-php5 php5-apcu -y

Stop Apache

user@WIKI01:~$ sudo service apache2 stop

Make sure Apache won’t start at startup cause we want Apache to be managed by Corosync

user@WIKI01:~$ sudo systemctl disable apache2

Show the current partitions
Note: The values may be different from those below.

user@WIKI01:~$ sudo fdisk -l

Disk /dev/sdb: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5d0538d3

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1  *      2048   499711   497664  243M 83 Linux
/dev/sda2       501758 16775167 16273410  7.8G  5 Extended
/dev/sda5       501760 16775167 16273408  7.8G 8e Linux LVM

Disk /dev/mapper/WIKI01--vg-root: 7.4 GiB, 7935623168 bytes, 15499264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/WIKI01--vg-swap_1: 376 MiB, 394264576 bytes, 770048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

As you see, /dev/sdb is not partitioned. We’re gonna create one big partition.

user@WIKI01:~$ sudo fdisk /dev/sdb

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): n <--- (To create a new partion)
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)

Select (default p): p <--- (Choose Primary)

Partition number (1-4, default 1): <--- (Press Enter)

First sector (2048-10485759, default 2048): <--- (Press Enter)

Last sector, +sectors or +size{K,M,G,T,P} (2048-10485759, default 10485759): <--- (Press Enter)

Created a new partition 1 of type 'Linux' and of size 5 GiB.

Command (m for help): w <-- (To accept and write the changes)

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

Run fdisk -l again, You should see a /dev/sdb1
Note: The values may be different from those below.

user@WIKI01:~$ sudo  fdisk -l

Disk /dev/sdb: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5715c1fe

Device     Boot Start      End  Sectors Size Id Type
/dev/sdb1        2048 10485759 10483712   5G 83 Linux

Disk /dev/sda: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5d0538d3

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1  *      2048   499711   497664  243M 83 Linux
/dev/sda2       501758 16775167 16273410  7.8G  5 Extended
/dev/sda5       501760 16775167 16273408  7.8G 8e Linux LVM

Disk /dev/mapper/WIKI01--vg-root: 7.4 GiB, 7935623168 bytes, 15499264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/WIKI01--vg-swap_1: 376 MiB, 394264576 bytes, 770048 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Create a physical volume of /dev/sdb1

user@WIKI01:~$ sudo pvcreate /dev/sdb1

Create a volume group

user@WIKI01:~$ sudo vgcreate vg_data /dev/sdb1

Create a logical volume of 500MB

user@WIKI01:~$ sudo lvcreate --name lv_data --size 500M vg_data

Install & Configure DRBD on WIKI01 & WIKI02

Install the DRBD utilities

user@WIKI01:~$ sudo apt-get install drbd8-utils -y

Load the DRBD kernel module:

user@WIKI01:~$ sudo modprobe drbd

To check if it is loaded. Output should be similar to this one:

user@WIKI01:~$ sudo lsmod | grep drbd
drbd                  318526  0
lru_cache              13032  1 drbd
libcrc32c              12426  1 drbd

Create a copy of the default DRBD config-file

user@WIKI01:~$ sudo cp /etc/drbd.conf /etc/drbd.conf_orig

Delete all the content of DRBD config-file

user@WIKI01:~$ sudo bash -c 'cat /dev/null > /etc/drbd.conf'

Open the drbd.conf file for editing. I use nano

user@WIKI01:~$ sudo nano /etc/drbd.conf

Replace the contents of drbd.conf with this configuration.
Note: This config-file may differ from you own environment. Replace the values that are specific to your configuration.

global { usage-count no; }
common { syncer { rate 100M; } }
resource Wiki {
        protocol C;
        startup {
                wfc-timeout  15;
                degr-wfc-timeout 60;
        }
        net {
                cram-hmac-alg sha1;
                shared-secret "secret";
        }
        on WIKI01 {
                device /dev/drbd0;
                disk /dev/vg_data/lv_data;
                address 10.90.0.5:7788;
                meta-disk internal;
        }
        on WIKI02 {
                device /dev/drbd0;
                disk /dev/vg_data/lv_data;
                address 10.90.0.6:7788;
                meta-disk internal;
        }
}

Create the DRBD resource “Wiki”

user@WIKI01:~$ sudo drbdadm create-md Wiki

Start the DRBD resource

user@WIKI01:~$ sudo drbdadm up Wiki

Create a directory /data

user@WIKI01:~$ sudo mkdir -p /data

Initialize DRBD on WIKI01

Start DRBD synchronization from WIKI01 (Primary) to WIKI02 (Secondary)

user@WIKI01:~$ sudo drbdadm -- --overwrite-data-of-peer primary all

You can monitor with a simple command.

user@WIKI01:~$ sudo cat /proc/drbd

Wait with the next step until sync is at 100%.
Note: Output when it’s still replicating

version: 8.4.3 (api:1/proto:86-101)
srcversion: 1A9F77B1CA5FF92235C2213
 0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:410596 nr:0 dw:0 dr:411508 al:0 bm:25 lo:0 pe:1 ua:0 ap:0 ep:1 wo:f oos:102348
        [===============>....] sync'ed: 80.8% (102348/511948)K
        finish: 0:00:06 speed: 16,384 (14,628) K/sec

Note: Output when it’s fully synced

version: 8.4.3 (api:1/proto:86-101)
srcversion: 1A9F77B1CA5FF92235C2213
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:511948 nr:0 dw:0 dr:512860 al:0 bm:32 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

Create a ext4 filesystem on /dev/drbd0

user@WIKI01:~$ sudo mkfs.ext4 /dev/drbd0

Create a mount from /dev/drbd0 to /data

user@WIKI01:~$ sudo mount /dev/drbd0 /data

Create a MediaWiki directory on /data

user@WIKI01:~$ sudo mkdir /data/mediawiki

Create a symlink from /var/www/html/mediawiki to /data/mediawiki

user@WIKI01:~$ sudo ln -s /data/mediawiki /var/www/html/mediawiki

Unmount the data directory

user@WIKI01:~$ sudo umount /data

Stop the DRBD “Wiki” resource

user@WIKI01:~$ sudo drbdadm down Wiki

Disable DRBD from autostart at startup

user@WIKI01:~$ sudo systemctl disable drbd

Initialize DRBD on WIKI02

Create here also a symlink from /var/www/html/mediawiki to /data/mediawiki

user@WIKI02:~$ sudo ln -s /data/mediawiki /var/www/html/mediawiki

Stop the DRBD “Wiki” resource

user@WIKI02:~$ sudo drbdadm down Wiki

Disable DRBD from autostart at startup

user@WIKI02:~$ sudo systemctl disable drbd

Install & Configure Corosync and Pacemaker on WIKI01 & WIKI02

First we need to add the repository to install Corosync and Pacemaker.

user@WIKI01:~$ sudo bash -c 'cat > /etc/apt/sources.list.d/jessie-backports.list << "EOF"
deb http://http.debian.net/debian jessie-backports main
EOF'

Update the repository and install Corosync and Pacemaker.

user@WIKI01:~$ sudo apt-get update && sudo apt-get install -t jessie-backports pacemaker crmsh -y

Stop Pacemaker service .

user@WIKI01:~$ sudo service pacemaker stop

Create a copy of the default Corosync config-file

user@WIKI01:~$ sudo cp /etc/corosync/corosync.conf /etc/corosync/corosync.conf_orig

Delete all the content of Corosync config-file

user@WIKI01:~$ sudo bash -c 'cat /dev/null > /etc/corosync/corosync.conf'

open the corosync.conf file for editing. I use nano

user@WIKI01:~$ sudo nano /etc/corosync/corosync.conf

Replace the contents of corosync.conf with this configuration.
Note: This config-file may differ from you own environment. Replace the values that are specific to your configuration.

totem {
    version: 2
    token: 3000
    token_retransmits_before_loss_const: 10
    join: 60
    consensus: 5000
    vsftype: none
    max_messages: 20
    clear_node_high_bit: yes
    secauth: off
    threads: 0
    rrp_mode: active

    transport: udpu
    interface {
        ringnumber: 0
        bindnetaddr: 10.40.0.0
        mcastport: 5405
    }
}

nodelist {
  node {
    ring0_addr: 10.40.0.15
    name: WIKI01
    nodeid: 1
  }
  node {
    ring0_addr: 10.40.0.16
    name: WIKI02
    nodeid: 2
  }
}

amf {
    mode: disabled
}

service {
     ver:       0
     name:      pacemaker
}

aisexec {
        user:   root
        group:  root
}

logging {
        fileline: off
        to_stderr: yes
        to_logfile: yes
        logfile: /var/log/corosync/corosync.log
        to_syslog: yes
        syslog_facility: daemon
        debug: off
        timestamp: on
        logger_subsys {
                subsys: AMF
                debug: off
                tags: enter|leave|trace1|trace2|trace3|trace4|trace6
        }
}

quorum {
  provider: corosync_votequorum
  two_node: 1
  wait_for_all: 1
  last_man_standing: 1
  auto_tie_breaker: 0
}

Restart Corosync to load the new configuration.

user@WIKI01:~$ sudo service corosync restart

Start Pacemaker.

user@WIKI01:~$ sudo service pacemaker start

Now that Corosync and Pacemaker is installed and configured on both servers let’s check if our cluster works.

user@WIKI01:~$ sudo crm status

The output should look something like this (if not, wait for 30 seconds, then run the command again):

Stack: corosync
Current DC: WIKI01 (version 1.1.15-e174ec8) - partition with quorum
Last updated: Sat Apr 29 14:20:32 2017          Last change: Sat Apr 29 14:20:06 2017 by hacluster via crmd on WIKI01

2 nodes and 0 resources configured

Online: [ WIKI01 WIKI02 ]

Full list of resources:

Configure Cluster Properties

Now we’re ready to configure the properties of Pacemaker.
Note: All Pacemaker (crm) commands can be run from either node server, as it automatically synchronizes all cluster-related changes across all member nodes.

First we want to disable STONITH (Shoot The Other Node In The Head).
Note: STONITH is the ability to remove faulty nodes if they no longer meets the cluster requirements.

user@WIKI01:~$ sudo crm configure property stonith-enabled=false

Second we want to ignore the quorum policy.
Note: This setting only applies to 2-node clusters.

user@WIKI01:~$ sudo crm configure property no-quorum-policy=ignore

Thirth and last we want to configure weight-points.
Note: Weight-points are useful when a node goes down and up, this configuration makes sure that the resources running on the other are kept there in case of flapping.

user@WIKI01:~$ sudo crm configure property default-resource-stickiness=100

Create Cluster Resources

Now that we have configured the properties we’re ready to create resources.
Open a Pacemaker shell

user@MYSQL01:~$ sudo crm

Enter configuration mode

crm(live)# configure

First: We start by monitoring pacemaker. If a node loses his network connection, we want to failover all resources to the other node.

crm(live)configure# primitive Ping-Gateway ocf:pacemaker:ping \
    params dampen="20s" multiplier="1000" \
    host_list="10.40.0.1" \
    op monitor interval="5s" timeout="60s" \
    op start interval="0" timeout="60s" \
    op stop interval="0" timeout="60s"
clone clone_Ping-Gateway Ping-Gateway \
    meta is-managed="true" target-role="Started"

Second: Add a Cluster-IP address for our MediaWiki cluster

crm(live)configure# primitive Cluster-IP ocf:heartbeat:IPaddr2 \
    params ip="10.40.0.17" cidr_netmask="24" \
    meta migration-threshold=2 \
    op monitor interval=20 \
    timeout=60 \
    on-fail=restart

Thirth: Add the DRBD “Wiki” resource

primitive DRBD-Wiki ocf:linbit:drbd \
    params drbd_resource="Wiki" \
    op start interval="0" timeout="240" \
    op stop interval="0" timeout="100" \
    op monitor role=Master interval=59s timeout=30s \
    op monitor role=Slave interval=60s timeout=30s

Fourth: Add the DRBD Filesystem.

primitive Data-FS ocf:heartbeat:Filesystem \
    params device=/dev/drbd0 \
    directory=/data \
    fstype=ext4 \
    op monitor interval="40s"

Fifth: Add the Apache webserver

primitive WebServer lsb:apache2 \
    meta migration-threshold=2 \
    op monitor interval=20 \
    timeout=60 \
    on-fail=restart

Now that we have added all our primitive components let’s add the rest of the configuration.

Next we need to tell Pacemaker to start a set of resources and consider the primitives in the set as having 2 states, Master and slave

ms ms_DRBD-Wiki DRBD-Wiki \
     meta master-max="1" \
     master-node-max="1" \
     clone-max="2" \
     clone-node-max="1" \
     notify="true"

Create a resource group for our cluster and add the Cluster-IP, Data-FS and WebServer
The order of resources is also the start en shutdown orderIn my case.
Startup: 1. Cluster-IP, 2. Data-FS, 3. WebSever
Shutdown : 1. WebServer, 2. Data-FS, 3. Cluster-IP

group MediaWiki-Cluster Cluster-IP Data-FS WebServer

Create a location rule which will failover resources to another node.

location DRBD-Master_Ping ms_DRBD-Wiki \
    rule $id="DRBD-Master_Ping-rule" -inf: not_defined pingd or pingd lte 0

Configure a colocation restraint, to specify that the MediaWiki-Cluster

colocation MediaWiki-Cluster_on_DRBD-Wiki inf: MediaWiki-Cluster ms_DRBD-Wiki:Master

The last step is startup order. The Master must be active before resource group can be started.

order MediaWiki-Cluster_after_DRBD-Wiki inf: ms_DRBD-Wiki:promote MediaWiki-Cluster:start

The configuration is now complete.
Let’s check if the config is done correctly

crm(live)configure# show

We shoud see the following output:
Note: This output may differ from you own environment.

node 1: WIKI01
node 2: WIKI02
primitive Cluster-IP IPaddr2 \
        params ip=10.40.0.17 cidr_netmask=24 \
        meta migration-threshold=2 \
        op monitor interval=20 timeout=60 on-fail=restart
primitive DRBD-Wiki ocf:linbit:drbd \
        params drbd_resource=Wiki \
        op start interval=0 timeout=240 \
        op stop interval=0 timeout=100 \
        op monitor role=Master interval=59s timeout=30s \
        op monitor role=Slave interval=60s timeout=30s
primitive Data-FS Filesystem \
        params device="/dev/drbd0" directory="/data" fstype=ext4 \
        op monitor interval=40s
primitive Ping-Gateway ocf:pacemaker:ping \
        params dampen=20s multiplier=1000 host_list=10.40.0.1 \
        op monitor interval=5s timeout=60s \
        op start interval=0 timeout=60s \
        op stop interval=0 timeout=60s
primitive WebServer lsb:apache2 \
        meta migration-threshold=2 \
        op monitor interval=20 timeout=60 on-fail=restart
group MediaWiki-Cluster Cluster-IP Data-FS WebServer
ms ms_DRBD-Wiki DRBD-Wiki \
        meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
clone clone_Ping-Gateway Ping-Gateway \
        meta is-managed=true target-role=Started
location DRBD-Master_Ping ms_DRBD-Wiki \
        rule -inf: not_defined pingd or pingd lte 0
order MediaWiki-Cluster_after_DRBD-Wiki inf: ms_DRBD-Wiki:promote MediaWiki-Cluster:start
colocation MediaWiki-Cluster_on_DRBD-Wiki inf: MediaWiki-Cluster ms_DRBD-Wiki:Master
property cib-bootstrap-options: \
        have-watchdog=false \
        dc-version=1.1.15-e174ec8 \
        cluster-infrastructure=corosync \
        cluster-name=debian \
        stonith-enabled=false \
        no-quorum-policy=ignore \
        default-resource-stickiness=100

If the config looks correctly you can commit the changes

crm(live)configure# commit

Exit the Pacemaker shell and check if the Cluster-IP is up-and-running.

crm(live)configure# exit
bye
user@WIKI01:~$ sudo crm status

We shoud see the following output:
Note: This output may differ from you own environment.

Stack: corosync
Current DC: WIKI01 (version 1.1.15-e174ec8) - partition with quorum
Last updated: Sat Apr 29 15:07:42 2017          Last change: Sat Apr 29 15:07:12 2017 by root via cibadmin on WIKI01

2 nodes and 7 resources configured

Online: [ WIKI01 WIKI02 ]

Full list of resources:

 Clone Set: clone_Ping-Gateway [Ping-Gateway]
     Started: [ WIKI01 WIKI02 ]
 Master/Slave Set: ms_DRBD-Wiki [DRBD-Wiki]
     Masters: [ WIKI01 ]
     Slaves: [ WIKI02 ]
 Resource Group: MediaWiki-Cluster
     Cluster-IP (ocf::heartbeat:IPaddr2):       Started WIKI01
     Data-FS    (ocf::heartbeat:Filesystem):    Started WIKI01
     WebServer  (lsb:apache2):  Started WIKI01

As you can see the ms_DRBD-WIKI “Master” is started on WIKI01
Note: It could take a while before all the resources are started

Install & Configure MediaWiki and MySQL

Find out which node is currently the “Master”. In my case WIKI01

user@WIKI01:~$ sudo crm status
Stack: corosync
Current DC: WIKI01 (version 1.1.15-e174ec8) - partition with quorum
Last updated: Sat Apr 29 15:07:42 2017          Last change: Sat Apr 29 15:07:12 2017 by root via cibadmin on WIKI01

2 nodes and 7 resources configured

Online: [ WIKI01 WIKI02 ]

Full list of resources:

 Clone Set: clone_Ping-Gateway [Ping-Gateway]
     Started: [ WIKI01 WIKI02 ]
 Master/Slave Set: ms_DRBD-Wiki [DRBD-Wiki]
     Masters: [ WIKI01 ]
     Slaves: [ WIKI02 ]
 Resource Group: MediaWiki-Cluster
     Cluster-IP (ocf::heartbeat:IPaddr2):       Started WIKI01
     Data-FS    (ocf::heartbeat:Filesystem):    Started WIKI01
     WebServer  (lsb:apache2):  Started WIKI01

Go to the /tmp directory and download MediaWiki

user@WIKI01:~$ cd /tmp && sudo wget https://releases.wikimedia.org/mediawiki/1.28/mediawiki-1.28.1.tar.gz

Extract and copy the content to /var/www/html/mediawiki

user@WIKI01:~$ sudo tar -xvzf /tmp/mediawiki-*.tar.gz
user@WIKI01:~$ sudo mv /tmp/mediawiki-1.28.1/* /var/www/html/mediawiki

Change the owner rights

user@WIKI01:~$ sudo chown www-data:www-data -R /var/www/html/mediawiki

SSH to your MySQL Cluster

user@WIKI01:~$ sudo ssh user@MYSQL_CLS

Login into mysql and create a database for our MediaWiki

user@MYSQL01:~$ sudo mysql -u root -p
mysql> CREATE DATABASE My_Wiki;

Create 2 users, one for each MediaWiki host. Replace <password> with your own.
Note: The values may be different than those below

GRANT INDEX, CREATE, SELECT, INSERT, UPDATE, DELETE, ALTER, LOCK TABLES on My_Wiki.* to 'WikiAdmin'@'10.40.0.15' IDENTIFIED BY '<password>';
GRANT INDEX, CREATE, SELECT, INSERT, UPDATE, DELETE, ALTER, LOCK TABLES on My_Wiki.* to 'WikiAdmin'@'10.40.0.16' IDENTIFIED BY '<password>';
FLUSH PRIVILEGES;
EXIT;

Open a Webbrowser and go to http://<MediaWiki Cluster-IP>/mediawiki/.
Follow the steps and install your MediaWiki
Note: Enter your MySQL IP or FQDN at Database host for HA.

Congratz! You have created a working HA MediaWiki.

Leave a Reply

Your email address will not be published. Required fields are marked *