Multimodal end-to-end autonomous driving
Xiao, Yi (Centre de Visió per Computador (Bellaterra, Catalunya))
Codevilla Moraes, Felipe (Centre de Visió per Computador (Bellaterra, Catalunya))
Gurram, Akhil (Huawei Munich Research Center)
Urfalioglu, Onay (Huawei Munich Research Center)
López Peña, Antonio M. (Universitat Autònoma de Barcelona. Departament de Ciències de la Computació)

Date:	2022
Description:	11 pàg.
Abstract:	A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e. g. , LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i. e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.
Grants:	Agencia Estatal de Investigación TIN2017-88709-R Agència de Gestió d'Ajuts Universitaris i de Recerca 2017/FI-B1-00162
Note:	Altres ajuts: Antonio M. Lopez acknowledges the financial support by ICREA under the ICREA Academia Program. We also thank the Generalitat de Catalunya CERCA Program, as well as its ACCIO agency
Rights:	Tots els drets reservats.
Language:	Anglès
Document:	Article ; recerca ; Versió acceptada per publicar
Subject:	Semantics ; Task analysis ; Laser radar ; Autonomous vehicles ; Cameras
Published in:	IEEE Transactions on Intelligent Transportation Systems, Vol. 23, issue 1 (Jan. 2022) , p. 537-547, ISSN 1558-0016

DOI: 10.1109/TITS.2020.3013234

Postprint 11 p, 1.7 MB

The record appears in these collections:
Articles > Research articles
Articles > Published articles