Recently I needed to replicate data between two Linux servers so I could mitigate potential hardware failure. To achieve this goal I used GlusterFS – an Open Source distributed file system. In this example I will use Debian Wheezy as I want to try more recent version of GlusterFS.
Assumptions
I will use server1 and server2 to create replicated volume. server3 and server4 to show how to extend it for size or replica count, server5 to migrate brick.
$ cat /etc/hosts | grep server
192.168.56.101 server1
192.168.56.102 server2
192.168.56.103 server3
192.168.56.104 server4
192.168.56.105 server5
I will use subdirectories of /var/export/ to store data internally and mirror /var/www directory.
Install GlusterFS
To use GlusterFS you need to install glusterfs-server package for server functionality and glusterfs-client (this package will be installed as dependency) to use created filesystem.
$ sudo apt-get install glusterfs-server
At the time of writing this post, the repository contains GlusterFS 3.2.7. You can skip it and install the latest version 3.3.0 using GlusterFS latest Debian packages link but you need to use Debian Wheezy because of libssl 1.0.0 dependency (Debian Squeeze contains libssl 0.9.8 version).
Create trusted storage pool
Create trusted network of storage servers, execute command on the first server:
server1# gluster peer probe server2
To add another server to trusted storage pool just probe it from any trusted server.
Verify peer status using command:
server1# gluster peer status
Number of Peers: 1
Hostname: server2
Uuid: aa251818-d4f6-47ab-a1d5-e9ced79d7daa
State: Peer in Cluster (Connected)
Create replicated volume between two servers
Create replicated volume www using two bricks. As stated above I will use directory /var/export/www on each server to store data internally.
# gluster volume create www replica 2 transport tcp server1:/var/export/www server2:/var/export/www
Set IP addresses of the clients which should be allowed to access this volume:
# gluster volume set www auth.allow 192.168.56.*
Start volume www:
# gluster volume start www
Check volume status using command:
# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*
Mounting volume
To mount volume on each server use command:
# mount -t glusterfs localhost:/www /var/www
I used localhost hostname so if other server is down it will mount anyway.
To automatically mount volume at boot on each server add similar fstab entry:
$ cat /etc/fstab | grep glusterfs
/etc/glusterd/vols/www/www-fuse.vol /var/www glusterfs defaults,_netdev 0 0
Extend trusted storage pool
Before I could continue I need to add three servers to trusted storage pool:
# gluster peer probe server3
# gluster peer probe server4
# gluster peer probe server5
Verify peer status using command:
server1# gluster peer status
Number of Peers: 4
Hostname: server2
Uuid: aa251818-d4f6-47ab-a1d5-e9ced79d7daa
State: Peer in Cluster (Connected)
Hostname: server3
Uuid: cfab1973-69a6-4d43-b3e1-a4dea34e4bcc
State: Peer in Cluster (Connected)
Hostname: server4
Uuid: d9e342ac-f503-4311-d2b9-cb4dcea38391
State: Peer in Cluster (Connected)
Hostname: server5
Uuid: 4d0abff1-32a8-9dcc-8263-a7cd7d4ae9cf
State: Peer in Cluster (Connected)
Extend volume for size – replicated-distributed volume
To extend amount of disk space available on volume www add a brick on server3 and server4 (multiple of the replica count):
# gluster volume add-brick www server3:/var/export/www server4:/var/export/www
This is now a replicated-distributed volume (2×2) and twice of disk space is available
# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Brick3: server3:/var/export/www
Brick4: server4:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*
Remove bricks
Remove bricks added in previous step. In case you created files on this volume you need to know that part of the data on will be lost.
# gluster volume remove-brick www server3:/var/export/www server4:/var/export/www
Remove directory /var/export/www on server3 and server4.
Extend volume for replica count
This is a bit tricky as for now there is no safe way to do it so backup is necessary.
Stop volume www:
# gluster volume stop www
Delete volume www:
# gluster volume delete www
Recreate volume but increase replica count to 4 and define all four bricks:
# gluster volume create www replica 4 transport tcp server1:/var/export/www server2:/var/export/www server3:/var/export/www server4:/var/export/www
Start volume www and set options:
# gluster volume start www
# gluster volume set www auth.allow 192.168.56.*
Check volume status:
# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Brick3: server3:/var/export/www
Brick4: server4:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*
Volume data should become available again but I cannot guarantee that.
Migrate brick
Bricks can be easily moved between servers. To move brick /var/export/www (volume www) from server4 to server5 execute commands below.
Start migration of server4:/var/export/www brick to server4:/var/export/www:
# gluster volume replace-brick server4:/var/export/www server5:/var/export/www start
Check status of above migration:
# gluster volume replace-brick server4:/var/export/www server5:/var/export/www status
After the migration is done you need to commit it:
# gluster volume replace-brick server4:/var/export/www server5:/var/export/www commit
Check volume status:
# gluster volume info www
Volume Name: www
Type: Replicate
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: server1:/var/export/www
Brick2: server2:/var/export/www
Brick3: server3:/var/export/www
Brick4: server5:/var/export/www
Options Reconfigured:
auth.allow: 192.168.56.*