At Converter App we process terabytes of data every day. When moving such large amounts of data, it is crucial to choose the right filesystems. Several distributed high-performance filesystems allow you to build a scalable computing cluster. In the following, we present our three favorites you can use to build your powerful computing cluster with ease. There are serial and parallel filesystems, but when you are interested in high I/O performance, you go for parallel file systems. As a result, we only consider such file systems. Here are the best parallel file systems in the year 2022 in comparison:
BeeGFS
BeeGFS is the leading open-source parallel file system developed by an active international community of developers. Initially, it was called FhGFS. In terms of performance, it is one of the leading file systems available, and the I/O speed scales with the size of your cluster. The installation and maintenance are done with the Linux command line, and there are guides to get started. There are repositories to get the BeeGFS packages for the leading Linux distributions.
There are three types of BeeGFS servers:
- Management Servers
- Meta Servers
- Storage Servers
The management server is responsible for the communication between clients and servers and also collects monitoring data. It can be considered the master node of the file system and is essential for its operation. The meta server stores the BeeGFS metadata. Each file has a corresponding metadata entry defining how the actual file contents are distributed across the storage servers.
During our tests, while setting up BeeGFS, the set-up went relatively smoothly. However, it took some time to understand the architecture and also find out the commands to use. Once set up, the BeeGFS were running smoothly. BeeGFS can be set up without necessarily formatting your disks on an existing cluster.
Pros:
- Excellent Performance
- 100% community driven and backed by a large community of developers
Cons:
- Needs some experience with the command line during setup
- The documentation is sometimes outdated and incomplete
- No graphical UI for the maintenance
Quobyte
Quobyte is a relatively new distributed file system developed by the company with the same name. It promises to be the only 100% software storage solution for enterprises, offering a simple installation done in minutes. There is an automated Linux installer, Kubernetes setup with Helm Chart, and also an instant Docker demo.
What are the minimum requirements for Quobyte?
To set up a minimal storage cluster you need at least four servers. Each of them should have a free disk with 250 GB of storage. It needs to be added to the disk hosting your file system since it will be formatted during the setup process. Furthermore, it is recommended to have flash drives with a storage capacity of 250 GB for the meta servers to boost the performance of the filesystem. However, this is optional, and you can also set up meta servers with conventional SSDs. There shouldn’t be any firewalls between the servers you want to combine. Like with BeeGFS, the servers hosting the file system can also be used for other tasks.
Testing Quobyte with a minimal cluster:
During our tests, Quobyte has been set up on a minimal cluster consisting of four servers with two SSD each. The method of choice was the Linux-based installation script. Before the installation, it is recommended to equip all servers with SSH keys so that you can log in without SSH keys.
When starting the installer you will be asked for the lists of your storage and client servers. After that, the installation went through to completion automatically. In the end, the script will print a link where you can access the graphical control panel of Quobyte in your browser.
Here, you need to complete two final steps:
- Format the free disks on your servers
- Once complete, create your first storage volume
The volume will show up on the previously defined client machines. It is easily possible to add additional clients after the installation.
The installation procedure was as simple as promised and did not take long but a few minutes.
How does the performance of Quobyte compare to BeeGFS?
To compare the performance, we created a minimal setup consisting of 2 Meta and two storage servers for each file system. We then wrote the same dataset 3 GB consisting of small files for each file system. The write rate was comparable with 30 GB/s for both. For reading back the data we reached about 50 MB/s. Here are the rates comparisons:
For Quobyte, the performance scales were linear with the number of Servers. If you double the number of servers, the speed will increase by a factor of two.
Pros:
- Straight-forward installation within minutes
- Scales linearly
- A graphical user interface
- There is a free version
Cons:
- Relatively new
- The documentation sometimes lacks crucial information
Amazon FSx
If you are working in the Amazon Cloud, do not hesitate to spend money on your high-performance file system. Amazon FSx is the best choice. While the two file systems presented before are mainly for Linux-based environments, Amazon offers a different version of FSx based on various underlying file systems:
- FSx for ONTAP using NetApp’s ONTAP file system.
- FSx can also be built on the popular Open ZFS file system.
- There is an FSx version built on Windows Server.
- Finally, there is an FSx version based on Lustre, a parallel distributed file system similar to BeeGFS.
The latter is the best choice when building a high-performance computing cluster consisting of Linux-based worker nodes. Like BeeGFS, Lustre is open source and available under the GNU General Public License. Setting up an FSx Lustre volume in the AWS console is easy and only requires a few clicks.
The minimum file size of a Volume is 1.2 TB which will incur monthly charges of about 200 USD depending on your region. You can choose between different speeds from 125 MB/s to 1 GB/s.
Is it possible to access a Lustre FSx volume outside the Amazon cloud?
In Principle, it is possible to access FSx Lustre from outside the Amazon cloud. However, you either require a VPN or a direct link to the Amazon cloud for doing so. Both setups will produce additional costs while the VPN is cheaper. The costs for the VPN depend on the outgoing traffic from the Amazon cloud.
Also, when mounting the file system from outside the cloud you will not reach the same performance unless you use a direct link. So the answer is yes, but when your computing cluster is located outside the Amazon Cloud we recommend either BeeGFS or Quobyte.
Pros:
- Volumes can be created dynamically from the AWS console
- Can be mounted from your Amazon cloud instances
- The file system is fully managed
Cons:
- A bit expensive
- Usage from outside the Amazon cloud produces additional costs and has performance trade-offs when not using a direct link to the Amazon cloud
