めもめも

このブログに記載の内容は個人の見解であり、必ずしも所属組織の立場、戦略、意見を代表するものではありません。

Using LevelDB over GlusterFS with libgfapi

I created a module with which you can use LevelDB[1] over GlusterFS _without_ FUSE client. This module uses libgfapi[2] which is a native API library for GlsuterFS. It allows applications to bypass the FUSE layer and access GlusterFS volumes directly. libgfapi gives performance improvement to applications using GlusterFS volumes.

Source code

1. My custom gfapi module for leveldb: util/env_gfapi.cc
2. autotools config patch for leveldb's srpm from EPEL: autotools_gfapi.patch

I tested it with the following packages.

leveldb-1.7.0-2.el6.src.rpm
glusterfs-api-3.4.1-3.el6.x86_64
glusterfs-api-devel-3.4.1-3.el6.x86_64

Quick build

The following is a quick step to build gfapi enabled leveldb's RPM on RHEL6.

1. Install some pre-req packages.

# curl http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-epel.repo > /etc/yum.repos.d/glusterfs-epel.repo
# yum install http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
# yum groupinstall "Development Tools"
# yum install snappy-devel glusterfs-api-devel

2. Download leveldb source from EPEL.

# yumdownloader leveldb --source
# rpm -ivh leveldb-*.src.rpm

3. Create the following patch to the spec file.

/tmp/leveldb.spec.patch

# diff -u leveldb.spec.orig leveldb.spec
--- leveldb.spec.orig	2013-12-01 13:16:38.561999731 +0900
+++ leveldb.spec	2013-12-01 13:44:17.712989695 +0900
@@ -16,6 +16,7 @@
 BuildRequires:	autoconf
 BuildRequires:	automake
 BuildRequires:	libtool
+BuildRequires:	glusterfs-api-devel
 
 
 %description
@@ -43,6 +44,14 @@
 
 
 %build
+#### Quick hack for gfapi
+rm -f /tmp/env_gfapi.cc
+rm -f /tmp/autotools_gfapi.patch
+curl https://gist.github.com/enakai00/7728581/raw/abf88f3dba38de29e05e8cffdb6719e4346a5080/env_gfapi.cc > /tmp/env_gfapi.cc
+curl https://gist.github.com/enakai00/7728588/raw/eb57712ae8245b1ab5e4399ae6346db7f3d13e03/autotools_gfapi.patch > /tmp/autotools_gfapi.patch
+cp -f /tmp/env_gfapi.cc util/env_gfapi.cc
+patch -p1 < /tmp/autotools_gfapi.patch
+####
 autoreconf -ivf
 %configure --disable-static --with-pic
 make %{?_smp_mflags}
@@ -59,7 +68,8 @@
 make check || true
 %else
 # x86, x86_64, armv5tel,  ppc, ppc64, ppc64v7 s390, and s390x are fine
-make check
+#make check
+echo "skipping check..."
 %endif

4. Apply it and build the package.

# cd ~/rpmbuild/SPECS
# patch -p0 < /tmp/leveldb.spec.patch
# rpmbuild -bb leveldb.spec --target=x86_64

Test It!

After building the package, you can use the regression test suite(i.e. "# make check") to test it.

1. Prepare GlutsterFS Volumes
I don't explain details of GlusterFS here..., I just assume you have prepared the following volume using some GlusterFS servers.

Server: gluster01, gluster02
Volume: testvol01

Before starting the test, you need to create a test directory /tmp just under the volume's root. It can be done as below on the GlusterFS server.

# mkdir /mnt/vol
# mount -t glusterfs localhost:/testvol01 /mnt/vol
# mkdir /mnt/vol/tmp
# umount /mnt/vol

2. Run the regression test suite. GlusterFS server/volume is specified with the environmental variables GF_SERVER and GF_VOLUME.

# cd ~/rpmbuild/BUILD/leveldb-1.7.0
# export GF_SERVER="gluster01"
# export GF_VOLUME="testvol01"
# make check

Note that Makefile is modifed to skip the "corruption" test module as it requires direct access to the filesystem. It is incompatible with gfapi.

Here's the running time comparison between "libgfapi enabled version" vs "original version over FUSE mount". You can see slight performance gain even though I omitted the performance optimization using mmap which is used in the original version. And as the regression test suite contains many non-IO intensive tests, there will be more advantage for libgfapi in IO intensive cases, probably.

libgfapi
===================
All 18 tests passed
===================
real 3m40.197s
user 1m15.885s
sys 0m24.672s

FUSE mount
===================
All 18 tests passed
===================
real 3m55.989s
user 0m36.328s
sys 0m12.129s