BeeOND ("BeeGFS On Demand") was developed to enable easy creation of one or multiple BeeGFS instances on the fly. This ability can be useful in many different use-cases, for example in cloud environments, but especially to create temporary work file systems. BeeOND is typically used to aggregate the performance and capacity of internal SSDs or hard disks in compute nodes for the duration of a compute job. This provides addtional performance and a very elegant way of burst buffering.
Nowadays, most HPC cluster systems use a global storage system based on a parallel file system on dedicated servers to enable high throughput. Compute nodes are often equipped (or can easily be equipped) with internal hard disks or SSDs, which could deliver an additional performance advantage. The problem with the internal drives in compute nodes is that they provide neither the advantage of a single name space across multiple machines nor the flexibility and performance of a shared parallel file system. BeeOND solves this problem by creating a shared parallel filesystem on a "per-job basis" across all compute nodes that are part of the particular compute job, exactly for the runtime of the job.
BeeOND can be used independent of whether the global shared cluster file system is based on BeeGFS or on other technology. BeeOND simply creates a new separate mountpoint. Any of the standard tools (like cp or rync) can be used to transfer data into and out of BeeOND, but the BeeOND package also contains a parallel copy tool to transfer data between BeeOND instances and another file system, such as your persitent global BeeGFS.
Due to the very simplified startup, it is easy to integrate BeeOND with workload managers, such as Torque or Slurm. Torque, for example, provides prologue and epilogue scripts, which will be executed on the first node assigned to a job. As BeeOND can start and stop new BeeGFS instances with just one single command, you can easily add it to these scripts to start BeeOND when a compute job starts and stop if when the job is finished. Please refer to the documentation of your workload manager for similar mechanisms.
The main advantages of the typical BeeOND use-case on compute nodes are:
- A very easy way to remove I/O load and possibly nasty I/O patterns from your persistent global file system. Temporary data created during the job runtime will never need to be moved to your global persistent file system, anyways. But even the data that should be preserved after the job end might be better stored to a BeeOND instance initially and then at the end can be copied to the persistent global storage completely sequentially in large chunks for maximum bandwidth.
- Applications that run on BeeOND do not "disturb" other users of the global parallel file system and in turn also got the performance of the BeeOND drives exclusively for themselfes without any influence by other users.
- Applications can complete faster, because with BeeOND, they can be running on SSDs (or maybe even a RAM-disk), while they might only be running on spinning disks on your normal persistent global file system. Combining the SSDs of multiple compute nodes not only gets you to high bandwith easily, it also gets you to a system that can handle very high IOPS.
- BeeOND does not add any costs for new servers, because you are simply using the compute nodes that you already have.
- You are turning the internal compute node drives, which might otherwise be useless for many distributed applications, into a shared parallel file system, which can easily be used for distributed applications.
- You get an easy chance to benefit from BeeGFS, even though your persistent global file system might not be based on BeeGFS.
BeeOND is available as standard package in the normal BeeGFS repositories and can be installed by using your distribution's package manager. For example, if running Red Hat, you can simply use yum on all nodes in your cluster:
$ ssh root@nodeXX $ yum install beeond
For operation, BeeOND needs the BeeGFS server and client components. Therefore, the respective packages are set as dependency and will be installed automatically.
If you want to use RDMA in the BeeOND environment you have to enable it for the client module as usual. This can be done by editing /etc/beegfs/beegfs-client-autobuild.conf as described in Building the Client Kernel Module.
The main ingredient of the BeeOND package is a script to start and stop a BeeGFS file system instance. This script is located in /opt/beegfs/sbin/beeond. A BeeOND instance can be controlled using "beeond start" and "beeond stop".
The easiest way to start a BeeOND instance is running beeond with the following set of parameters:
[root@fhgfs01 ~]# beeond start -n nodefile -d /data/beeond -c /mnt/beeond
Description of start parameters:
- nodefile: A file containing all hostnames to run BeeOND on (one host per line). All hosts in this file will become clients and servers for this new file system instance.
- /data/beeond: Should be substituted with a path, in which BeeOND will save its raw data on each node.
- /mnt/beeond: Should be substituted with the path where the BeeOND instance shall be mounted on each node.
To shut a BeeOND instance down, you only need to use the "stop" mode of the script, similar to startup:
$ beeond stop -n nodefile -L -d
Description of stop parameters:
- nodefile: A file containing all hostnames that were used on startup of the BeeOND instance.
- -L: Deletes all log files that were created by this BeeOND instance.
- -d: Deletes the raw data that was written by this BeeOND instance.
For a more detailed information and additional parameters, please see:
$ beeond -h
The tool "beeond-cp" can be used to perform a parallel stage-in and stage-out of the data you need. If we assume, that you need two working sets, which are located in /scratch/first and /scratch/second you can use the following command to copy the data into /mnt/beeond:
$ beeond-cp copy -n nodefile /scratch/first /scratch/second /mnt/beeond
Description of copy parameters:
- nodefile: A file containing all hostnames to run the distributed copy on (typically the same nodefile that was used for "beeond start").
- /scratch/first and /scratch/second: The source directories to be copied (you can use one or multiple files or directories as source here).
- /mnt/beeond: The target directory for the parallel copy.
For a more detailed information, please see:
$ beeond-cp -h
Back to User Guide - Table of Contents