Seeing is believing...
This video is created with the help of IIJ Inc. They kindly offered me servers in their cloud datacenter :-)
Software prereqs
We use GlusterFS3.3 on RHEL6.2.
[root@gluster01 ~]# rpm -qa | grep gluster glusterfs-3.3.0-1.el6.x86_64 glusterfs-fuse-3.3.0-1.el6.x86_64 glusterfs-server-3.3.0-1.el6.x86_64 glusterfs-geo-replication-3.3.0-1.el6.x86_64
You can find the latest GlusterFS3.3 RHEL RPMs here.
Overall configuration
Replicate volume "vol01" on the master cluster to volume "vol01_slave" on the slave cluster.
- 4nodes GlusterFS Cluster(Replication Master)
[gluster01]----[gluster02]----[gluster03]----[gluster04]
|
| Geo Replication
V
[gluster05]----[gluster06]----[gluster07]----[gluster08]
- 4nodes GlusterFS Cluster(Replication Slave)We use the same configuration (4nodes with 2replicas) for vol01 and vol01_slave for the sake of simplicity. If you want, you can use asymmetric volume configuration, too.
[root@gluster01 ~]# gluster vol info Volume Name: vol01 Type: Distributed-Replicate Volume ID: 10d3504b-7111-467d-8d4f-d25f0b504df6 Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gluster01:/data/brick01 Brick2: gluster02:/data/brick01 Brick3: gluster03:/data/brick01 Brick4: gluster04:/data/brick01 Options Reconfigured: geo-replication.indexing: on [root@gluster01 ~]# ssh gluster05 gluster vol info Volume Name: vol01_slave Type: Distributed-Replicate Volume ID: 4434468b-a23c-423d-b42d-066d6052684f Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: gluster05:/data/brick01 Brick2: gluster06:/data/brick01 Brick3: gluster07:/data/brick01 Brick4: gluster08:/data/brick01
Files are transferred between specific nodes on master and slave. In this case, we chose gluster01 on the master and gluster05 on the slave. File transfer is secured with the SSH tunneling.
Preparation on Slave
The following is done on the salve (gluster05).
First, because of the known issue, you have to set SELinux to the permissive mode on the slave.
[root@gluster05 ~]# setenforce 0 [root@gluster05 ~]# sed -i 's/SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
As I dropped the result of my investigation in Bugzilla, this would be resolved soon, hopefully.
Then, create a user and group to accept the SSH connection from the master. The directory "/var/mountbroker-root" is necessary for the mountbroker mechanism which allows non-privileged users to mount the volume.
[root@gluster05 ~]# groupadd georep [root@gluster05 ~]# useradd -G georep georep01 [root@gluster05 ~]# passwd georep01 [root@gluster05 ~]# mkdir /var/mountbroker-root [root@gluster05 ~]# chmod 711 /var/mountbroker-root/
Add the options to the configuration file (/etc/glusterfs/glusterd.vol) as below.
[root@gluster05 ~]# cat /etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option mountbroker-root /var/mountbroker-root
option mountbroker-geo-replication.georep01 vol01_slave
option geo-replication-log-group georep
end-volume
When accepting multiple volumes with the same user, you can specify multiple volumes.
option mountbroker-geo-replication.georep01 vol01_slave,vol02_slaveWhen you want to use separate users for different volumes, you need to add multiple entries.
option mountbroker-geo-replication.georep01 vol01_slave option mountbroker-geo-replication.georep02 vol02_slave
Finally, restart the glusterd service to reflect the configuration changes.
[root@gluster05 ~]# service glusterd restart Stopping glusterd: [ OK ] Starting glusterd: [ OK ] [root@gluster05 ~]# service glusterd status glusterd (pid 1590) is running...
At this point, the daemon creates the directory "/var/mountbroker-root/mb_hive". Without it, GeoReplication definitely fails. If you accidentally removed it, restart the glusterd service to recreate it.
Preparation on Master
The following is done on the master (gluster01).
Configure the public key authentication against "georep01@gluster05".
[root@gluster01 ~]# ssh-keygen -P "" [root@gluster01 ~]# ssh-copy-id georep01@gluster05
Add the "command=" option to the slave's authorized_keys.
[root@gluster05 ~]# cat /home/georep01/.ssh/authorized_keys command="/usr/libexec/glusterfs/gsyncd" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAs5K71oRlcr+O+l7Okwgw・・・
This restricts the remote command execution only to "gsyncd". This is mandatory for GerReplication to work, not a security option.
Now you can start the GeoReplication.
[root@gluster01 ~]# gluster vol geo-replication vol01 georep01@gluster05::vol01_slave start Starting geo-replication session between vol01 & georep01@gluster05::vol01_slave has been successful [root@gluster01 ~]# gluster vol geo-replication vol01 status MASTER SLAVE STATUS -------------------------------------------------------------------------------- vol01 ssh://georep01@gluster05::vol01_slave starting...
"georep01@gluster05::vol01_slave" is an abbreviation of "ssh://georep01@gluster05::vol01_slave". They are interchangeable.
Waiting for a while, the status will become "OK".
[root@master01 ~]# gluster vol geo-replication vol01 status MASTER SLAVE STATUS -------------------------------------------------------------------------------- vol01 ssh://georep01@gluster05::vol01_slave OK
You did it! GeoReplication is now working for you. Files stored in vol01 are asynchronously replicated to vol01_slave.
[root@gluster01 ~]# mount -t glusterfs localhost:/vol01 /mnt [root@gluster01 ~]# for num in $(seq 0 9); do cp /boot/vmlinuz-2.6.32-220.13.1.el6.x86_64 /mnt/file$num;done [root@gluster05 ~]# mount -t glusterfs localhost:/vol01_slave /mnt [root@gluster05 ~]# ls -l /mnt Total 38520 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file0 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file1 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file2 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file3 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file4 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file5 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file6 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file7 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file8 -rwxr-xr-x. 1 root georep01 3941136 May 9 16:18 2012 file9
