Home // INFOCOMP 2021, The Eleventh International Conference on Advanced Communications and Computation // View article

An API to Include HPC Resources in Workflow Systems

Authors:
Sven Bingert
Christian Köhler
Hendrik Nolte
Waqar Alamgir

Keywords: HPC, automation, RESTful API, workflow en- gine, data management, provenance, data lake

Abstract:
The demand for processing power by modern data analyses is continuously increasing. High-Performance- Computing (HPC) resources can help but the standard process is for users to log in to use the HPC systems which is often complicated and not well suited for the integration in workflows. In order to bridge the gap between external workflow tools and the usage of HPC resources, we designed and implemented an application interface. This API allows workflow systems to submit HPC jobs along with required artefacts to the queuing system without a direct login of the user. The presented API regards the required safety regulations by ensuring the identity of authorised external workflow systems, as well as the executing HPC systems with a token-based authentication model. In this paper we describe the design of the API and present three use-cases. In the data lake use-case, a novel technique for provenance auditing without runtime overhead is presented which is particularly well suited for HPC systems.

Pages: 15 to 20

Copyright: Copyright (c) IARIA, 2021

Publication date: May 30, 2021

Published in: conference

ISSN: 2308-3484

ISBN: 978-1-61208-865-5

Location: Valencia, Spain

Dates: from May 30, 2021 to June 3, 2021