BeeGFS as the Hadoop File System


Hadoop can be configured to use BeeGFS as its distributed file system, as a more convenient and faster alternative than using HDFS. This page explains how to implement and test such configuration.

Table of Contents (Page)

  1. Overview
  2. Using the BeeGFS Hadoop Connector
  3. Using the BeeGFS POSIX Interface
  4. Testing Hadoop and BeeGFS
 

Overview


There are 2 ways of integrating Hadoop and BeeGFS.


The procedures to configure Hadoop in each scenario are presented below.


Using the BeeGFS Hadoop Connector


The following procedure has been tested for Hadoop 2.5.2, 2.6.2, and 2.7.2.


See below an example of the file core-site.xml containing the BeeGFS connector configuration.
    <configuration>
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/data/hadoop-tmp/</value>
        </property>
   
        <property>
            <name>fs.default.name</name>
            <value>viewfs:///</value>
        </property>
   
        <property>
            <name>fs.AbstractFileSystem.beegfs.impl</name>
            <value>com.beegfs.BeeGFS</value>
        </property>
   
        <property>
            <name>fs.beegfs.node-name-suffix</name>
            <value>.global.cluster</value>
        </property>
   
        <property>
            <name>fs.viewfs.mounttable.default.link./tmp</name>
            <value>beegfs:/mnt/beegfs/hadoop/tmp</value>
        </property>
       
        <property>
            <name>fs.viewfs.mounttable.default.link./user</name>
            <value>beegfs:/mnt/beegfs/hadoop/user</value>
        </property>
    </configuration>



Using the BeeGFS POSIX Interface


The following procedure has been tested for Hadoop 2.7.2.


See below an example of the file core-site.xml containing the POSIX-based configuration.
    <configuration>
        <property>
            <name>hadoop.tmp.dir</name>
            <value>/data/hadoop-tmp/</value>
        </property>
   
        <property>
            <name>fs.default.name</name>
            <value>viewfs:///</value>
        </property>
   
        <property>
            <name>fs.AbstractFileSystem.file.impl</name>
            <value>org.apache.hadoop.fs.local.LocalFs</value>
        </property>
   
        <property>
            <name>fs.viewfs.mounttable.default.link./tmp</name>
            <value>file:/mnt/beegfs/hadoop/tmp</value>
        </property>
   
        <property>
            <name>fs.viewfs.mounttable.default.link./user</name>
            <value>file:/mnt/beegfs/hadoop/user</value>
        </property>
    </configuration>



Testing Hadoop and BeeGFS


After following one of the procedures above, you should be able to test the system, as follows.

Valid XHTML :: Valid CSS: :: Powered by WikkaWiki