Home // SCALABILITY 2024, The First International Conference on Systems Scalability and Expandability // View article

A Quantitative and Qualitative Comparison of Machine Learning Inference Frameworks

Authors:
Egi Brako
Julian Kunkel
Jonathan Decker

Keywords: machine learning; artificial intelligence; inference engines

Abstract:
As AI continues to advance and impact diverse fields, ensuring universal access to its abilities becomes increasingly crucial. To make AI models accessible to users, they must be deployed to process inference requests. We conducted qualitative and quantitative analyses of popular open-source serving frameworks by evaluating their performance on three Machine Learning tasks. This research aims to shed more light on the frameworks' respective strengths and weaknesses, consequently addressing the challenges posed by the process of selecting a method of serving the models. The qualitative comparison is carried out by taking into account the subjective characteristics of each framework and scoring them on a number scale. We then use Locust to run load-tests on these frameworks, analyse their quantitative results, and compare them with each other. Our results find that PyTorch TorchServe is the overall best-performing framework, consistently surpassing the other two in our performance test. We find that some platforms have issues handling more complex models, showing incapabilities for handling specific Machine Learning tasks. Our findings show significant differences among the frameworks, contributing valuable insights for developers and researchers in selecting the most suitable framework serving Machine Learning models.

Pages: 7 to 13

Copyright: Copyright (c) IARIA, 2024

Publication date: November 17, 2024

Published in: conference

ISBN: 978-1-68558-216-6

Location: Valencia, Spain

Dates: from November 17, 2024 to November 21, 2024