I created a module with which you can use LevelDB[1] over GlusterFS _without_ FUSE client. This module uses libgfapi[2] which is a native API library for GlsuterFS. It allows applications to bypass the FUSE layer and access GlusterFS volumes directly. libgfapi gives performance improvement to applications using GlusterFS volumes.
- [1] leveldb
- [2] glusterfs/api
Source code
1. My custom gfapi module for leveldb: util/env_gfapi.cc
2. autotools config patch for leveldb's srpm from EPEL: autotools_gfapi.patch
I tested it with the following packages.
leveldb-1.7.0-2.el6.src.rpm glusterfs-api-3.4.1-3.el6.x86_64 glusterfs-api-devel-3.4.1-3.el6.x86_64
Quick build
The following is a quick step to build gfapi enabled leveldb's RPM on RHEL6.
1. Install some pre-req packages.
# curl http://download.gluster.org/pub/gluster/glusterfs/LATEST/RHEL/glusterfs-epel.repo > /etc/yum.repos.d/glusterfs-epel.repo # yum install http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm # yum groupinstall "Development Tools" # yum install snappy-devel glusterfs-api-devel
2. Download leveldb source from EPEL.
# yumdownloader leveldb --source # rpm -ivh leveldb-*.src.rpm
3. Create the following patch to the spec file.
/tmp/leveldb.spec.patch
# diff -u leveldb.spec.orig leveldb.spec --- leveldb.spec.orig 2013-12-01 13:16:38.561999731 +0900 +++ leveldb.spec 2013-12-01 13:44:17.712989695 +0900 @@ -16,6 +16,7 @@ BuildRequires: autoconf BuildRequires: automake BuildRequires: libtool +BuildRequires: glusterfs-api-devel %description @@ -43,6 +44,14 @@ %build +#### Quick hack for gfapi +rm -f /tmp/env_gfapi.cc +rm -f /tmp/autotools_gfapi.patch +curl https://gist.github.com/enakai00/7728581/raw/abf88f3dba38de29e05e8cffdb6719e4346a5080/env_gfapi.cc > /tmp/env_gfapi.cc +curl https://gist.github.com/enakai00/7728588/raw/eb57712ae8245b1ab5e4399ae6346db7f3d13e03/autotools_gfapi.patch > /tmp/autotools_gfapi.patch +cp -f /tmp/env_gfapi.cc util/env_gfapi.cc +patch -p1 < /tmp/autotools_gfapi.patch +#### autoreconf -ivf %configure --disable-static --with-pic make %{?_smp_mflags} @@ -59,7 +68,8 @@ make check || true %else # x86, x86_64, armv5tel, ppc, ppc64, ppc64v7 s390, and s390x are fine -make check +#make check +echo "skipping check..." %endif
4. Apply it and build the package.
# cd ~/rpmbuild/SPECS # patch -p0 < /tmp/leveldb.spec.patch # rpmbuild -bb leveldb.spec --target=x86_64
Test It!
After building the package, you can use the regression test suite(i.e. "# make check") to test it.
1. Prepare GlutsterFS Volumes
I don't explain details of GlusterFS here..., I just assume you have prepared the following volume using some GlusterFS servers.
Server: gluster01, gluster02
Volume: testvol01
Before starting the test, you need to create a test directory /tmp just under the volume's root. It can be done as below on the GlusterFS server.
# mkdir /mnt/vol # mount -t glusterfs localhost:/testvol01 /mnt/vol # mkdir /mnt/vol/tmp # umount /mnt/vol
2. Run the regression test suite. GlusterFS server/volume is specified with the environmental variables GF_SERVER and GF_VOLUME.
# cd ~/rpmbuild/BUILD/leveldb-1.7.0 # export GF_SERVER="gluster01" # export GF_VOLUME="testvol01" # make check
Note that Makefile is modifed to skip the "corruption" test module as it requires direct access to the filesystem. It is incompatible with gfapi.
Here's the running time comparison between "libgfapi enabled version" vs "original version over FUSE mount". You can see slight performance gain even though I omitted the performance optimization using mmap which is used in the original version. And as the regression test suite contains many non-IO intensive tests, there will be more advantage for libgfapi in IO intensive cases, probably.
libgfapi
===================
All 18 tests passed
===================
real 3m40.197s
user 1m15.885s
sys 0m24.672s
FUSE mount
===================
All 18 tests passed
===================
real 3m55.989s
user 0m36.328s
sys 0m12.129s