読者です 読者をやめる 読者になる 読者になる

めもめも

このブログに記載の内容は個人の見解であり、必ずしも所属組織の立場、戦略、意見を代表するものではありません。

Quick Start to GeoReplication with GlusterFS3.3

Seeing is believing...

This video is created with the help of IIJ Inc. They kindly offered me servers in their cloud datacenter :-)

Software prereqs

We use GlusterFS3.3 on RHEL6.2.

[root@gluster01 ~]# rpm -qa | grep gluster
glusterfs-3.3.0-1.el6.x86_64
glusterfs-fuse-3.3.0-1.el6.x86_64
glusterfs-server-3.3.0-1.el6.x86_64
glusterfs-geo-replication-3.3.0-1.el6.x86_64

You can find the latest GlusterFS3.3 RHEL RPMs here.

Overall configuration

Replicate volume "vol01" on the master cluster to volume "vol01_slave" on the slave cluster.

- 4nodes GlusterFS Cluster(Replication Master)
[gluster01]----[gluster02]----[gluster03]----[gluster04]
    |
    | Geo Replication
    V
[gluster05]----[gluster06]----[gluster07]----[gluster08]
- 4nodes GlusterFS Cluster(Replication Slave)

We use the same configuration (4nodes with 2replicas) for vol01 and vol01_slave for the sake of simplicity. If you want, you can use asymmetric volume configuration, too.

[root@gluster01 ~]# gluster vol info
Volume Name: vol01
Type: Distributed-Replicate
Volume ID: 10d3504b-7111-467d-8d4f-d25f0b504df6
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gluster01:/data/brick01
Brick2: gluster02:/data/brick01
Brick3: gluster03:/data/brick01
Brick4: gluster04:/data/brick01
Options Reconfigured:
geo-replication.indexing: on

[root@gluster01 ~]# ssh gluster05 gluster vol info
Volume Name: vol01_slave
Type: Distributed-Replicate
Volume ID: 4434468b-a23c-423d-b42d-066d6052684f
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gluster05:/data/brick01
Brick2: gluster06:/data/brick01
Brick3: gluster07:/data/brick01
Brick4: gluster08:/data/brick01

Files are transferred between specific nodes on master and slave. In this case, we chose gluster01 on the master and gluster05 on the slave. File transfer is secured with the SSH tunneling.

Preparation on Slave

The following is done on the salve (gluster05).

First, because of the known issue, you have to set SELinux to the permissive mode on the slave.

[root@gluster05 ~]# setenforce 0
[root@gluster05 ~]# sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

As I dropped the result of my investigation in Bugzilla, this would be resolved soon, hopefully.

Then, create a user and group to accept the SSH connection from the master. The directory "/var/mountbroker-root" is necessary for the mountbroker mechanism which allows non-privileged users to mount the volume.

[root@gluster05 ~]# groupadd georep
[root@gluster05 ~]# useradd -G georep georep01
[root@gluster05 ~]# passwd georep01
[root@gluster05 ~]# mkdir /var/mountbroker-root
[root@gluster05 ~]# chmod 711 /var/mountbroker-root/

Add the options to the configuration file (/etc/glusterfs/glusterd.vol) as below.

[root@gluster05 ~]# cat /etc/glusterfs/glusterd.vol 
volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option transport.socket.read-fail-log off

    option mountbroker-root /var/mountbroker-root 
    option mountbroker-geo-replication.georep01 vol01_slave
    option geo-replication-log-group georep
end-volume

When accepting multiple volumes with the same user, you can specify multiple volumes.

option mountbroker-geo-replication.georep01 vol01_slave,vol02_slave

When you want to use separate users for different volumes, you need to add multiple entries.

option mountbroker-geo-replication.georep01 vol01_slave
option mountbroker-geo-replication.georep02 vol02_slave

Finally, restart the glusterd service to reflect the configuration changes.

[root@gluster05 ~]# service glusterd restart
Stopping glusterd:                                         [  OK  ]
Starting glusterd:                                         [  OK  ]
[root@gluster05 ~]# service glusterd status
glusterd (pid  1590) is running...

At this point, the daemon creates the directory "/var/mountbroker-root/mb_hive". Without it, GeoReplication definitely fails. If you accidentally removed it, restart the glusterd service to recreate it.

Preparation on Master

The following is done on the master (gluster01).

Configure the public key authentication against "georep01@gluster05".

[root@gluster01 ~]# ssh-keygen -P ""
[root@gluster01 ~]# ssh-copy-id georep01@gluster05

Add the "command=" option to the slave's authorized_keys.

[root@gluster05 ~]# cat /home/georep01/.ssh/authorized_keys 
command="/usr/libexec/glusterfs/gsyncd" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs5K71oRlcr+O+l7Okwgw・・・

This restricts the remote command execution only to "gsyncd". This is mandatory for GerReplication to work, not a security option.

Now you can start the GeoReplication.

[root@gluster01 ~]# gluster vol geo-replication vol01 georep01@gluster05::vol01_slave start
Starting geo-replication session between vol01 & georep01@gluster05::vol01_slave has been successful

[root@gluster01 ~]# gluster vol geo-replication vol01 status
MASTER               SLAVE                                              STATUS    
--------------------------------------------------------------------------------
vol01                ssh://georep01@gluster05::vol01_slave              starting...

"georep01@gluster05::vol01_slave" is an abbreviation of "ssh://georep01@gluster05::vol01_slave". They are interchangeable.

Waiting for a while, the status will become "OK".

[root@master01 ~]# gluster vol geo-replication vol01 status
MASTER               SLAVE                                              STATUS    
--------------------------------------------------------------------------------
vol01                ssh://georep01@gluster05::vol01_slave              OK    

You did it! GeoReplication is now working for you. Files stored in vol01 are asynchronously replicated to vol01_slave.

[root@gluster01 ~]# mount -t glusterfs localhost:/vol01 /mnt
[root@gluster01 ~]# for num in $(seq 0 9); do cp /boot/vmlinuz-2.6.32-220.13.1.el6.x86_64 /mnt/file$num;done

[root@gluster05 ~]# mount -t glusterfs localhost:/vol01_slave /mnt
[root@gluster05 ~]# ls -l /mnt
Total 38520
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file0
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file1
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file2
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file3
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file4
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file5
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file6
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file7
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file8
-rwxr-xr-x. 1 root georep01 3941136  May  9 16:18 2012 file9