Home // International Journal On Advances in Software, volume 18, numbers 1 and 2, 2025 // View article

Scalable Software Distribution for HPC-Systems with Software Pools Using MPI and File Systems in User Space

Authors:
Jakob Dieterle
Hendrik Nolte
Julian Kunkel

Keywords: file distribution, optimized reading, containerization

Abstract:
Despite the increasing computing power of high- performance computing (HPC) systems, complex tasks on large- scale clusters can still be hindered by significant waiting times when loading large software packages and dependencies. These delays are often caused by network bandwidth bottlenecks, which can severely impact application performance. To address this challenge, this paper presents a new way of distributing software in HPC systems. Our software pools can hold whole software stacks in a single file, while our implemented tools can distribute software pools efficiently to large clusters while reducing bandwidth usage to a minimum. Software pools offer additional advantages, such as portability, reproducibility, and security, while seamlessly integrating into existing environments using Lmod.

Pages: 97 to 105

Copyright: Copyright (c) to authors, 2025. Used with permission.

Publication date: June 30, 2025

Published in: journal

ISSN: 1942-2628