Home // CLOUD COMPUTING 2021, The Twelfth International Conference on Cloud Computing, GRIDs, and Virtualization // View article

Take Me to the Clouds Above: Bridging On Site HPC with Clouds for Capacity Workloads

Authors:
Jay Lofstead
Dmitry Duplyakin

Keywords: Cloud; HPC; Cloud Bursting; schedulers;

Abstract:
Sites with limited compute cluster capacity aimed at supporting large-scale applications of scientific and parallel computing must deal with the additional demand for small-scale jobs---in many cases, single-node and coming in large volumes---that help further develop the capabilities of large-scale applications as well as run simple data analysis tasks. Many of these analysis tasks and other small-scale jobs are run on High Performance Computing (HPC) systems because they offer familiar environments, making future scaling convenient, or because the proximity to the input or output data sets is important. When these small-scale jobs explode in count and create significant competition for resources, ensuring that the most effective use of the cluster resources is achieved requires either non-trivial scheduler tuning for better management of job mixes or adoption of entirely different models for management of computing environments. We claim that a hybrid ``on-site HPC + cloud'' model can offer the best of both worlds. Thus, small jobs should target cloud resources in cases where they can be offloaded to clouds without significant penalties on performance, and, at the same time, large-scale application runs can better utilize HPC clusters, achieving lower wait times and increasing job throughput. This is not without challenges.

Pages: 54 to 59

Copyright: Copyright (c) The Government of USA, 2021. Used by permission to IARIA.

Publication date: April 18, 2021

Published in: conference

ISSN: 2308-4294

ISBN: 978-1-61208-845-7

Location: Porto, Portugal

Dates: from April 18, 2021 to April 22, 2021