Home // International Journal On Advances in Systems and Measurements, volume 16, numbers 1 and 2, 2023 // View article

A Multi-modal AI Approach for Intuitively Instructable Autonomous Systems

Authors:
Ferran Gebellí Guinjoan
Matthias Hutsebaut-Buysse
Gorjan Radevski
Hugo Van Hamme
Erwin Rademakers
Anil Kumar Chavali
Kevin Mets
Tom De Schepper
Steven Latré
Erik Mannens
Tinne Tuytelaars
Abdellatif Bey Temsamani

Keywords: AI based autonomous systems; multi-modal AI; natural language processing; deep learning; neural networks; reinforcement learning

Abstract:
We present a multi-modal AI framework to intuitively instruct and control Automated Guided Vehicles. We define a general multi-modal AI architecture, which has a loose coupling between three different AI modules, including spoken language understanding, visual perception and Reinforcement Learning navigation. We use the same multi-modal architecture for two different use cases implemented in two different platforms: an off-road vehicle, which can pick objects, and an indoor forklift that performs automated warehouse inventory. We show how the proposed architecture can be used for a wide range of tasks and can be implemented in different hardware, demonstrating a high degree of modularity.

Pages: 1 to 13

Copyright: Copyright (c) to authors, 2023. Used with permission.

Publication date: June 30, 2023

Published in: journal

ISSN: 1942-261x