Home // SCALABILITY 2024, The First International Conference on Systems Scalability and Expandability // View article

Scalable Software Distribution for HPC-Systems Using MPI-Based File Systems in User Space

Authors:
Jakob Dieterle
Hendrik Nolte
Julian Kunkel

Keywords: data distribution, optimized reading, one-sided communication

Abstract:
Even on state-of-the-art high-performance computing systems, data-intensive tasks that are running on a lot of nodes can encounter waiting times when accessing large files. This can be caused by a bottleneck in the network bandwidth. To this end, this paper aims to develop a filesystem that utilizes inter-node communication to more efficiently distribute large files between nodes and reduce the bandwidth usage on the network. The presented file system was implemented using the Filesystem in Userspace interface and the Message Passing Interface. During evaluation, it showed promising performance compared to a slower native file system but did not reach the performance level of an optimized file system. A refined version was able to outperform the slower native file system with 16 nodes by 8%, achieving a bandwidth-reduction to latency-increase ratio of approximately 22.

Pages: 14 to 20

Copyright: Copyright (c) IARIA, 2024

Publication date: November 17, 2024

Published in: conference

ISBN: 978-1-68558-216-6

Location: Valencia, Spain

Dates: from November 17, 2024 to November 21, 2024