Continual Learning Framework for Opponent Detection in autonomous car races

Autonomous Driving Systems stand on their ability to understand the world around them, through Perception algorithms: in this work, it has been investigated how Object Detection can impact said systems, leveraging images captured through the on-board camera of the RoboRacer platform. The realization of this system, has be slowed down by two main problems: first, the lack of publicly available dataset to let the model train and learn on, and second, the form in which these data can be acquired. This last phenomenon is problematic, because it goes against one of the main requirements for data that has to be used in Convolutional Neural Networks: the i.i.d. (Independent and Identically Distributed) assumption. New data is usually available in sessions and streams of images, all linked by the environment it is recorded in. All these requirements move toward an approach that differs from the classical learning, called Incremental or Continual Learning. The first part of the work is represented by the creation of a novel and customizable dataset, well suited for welcoming new sessions and streams of data even after it has already approached training. This is not however all this thesis contributes to: a complete framework has been built that allows the actual addition of new data and then handles it in such a way to let any user of the project contribute and also design new experiments to evaluate the behavior of a network when trained on a different composition of available data and allows to consider situations where fine tuning is a possible application. This last situation is the most similar to the Continual Learning one: this type of Learning's main problem is Catastrophic Forgetting, that is the tendency of a model to forget old tasks and environments when tuned on new ones. Based on this, the work includes a section where the framework is employed to evaluate the performance of models when they have to handle new data. Three main situations have been investigated, like a full on retraining, starting from scratch, a naive fine tuning and one of the main approaches to limit said forgetting: the experience replay. It aims to add part of the old data to the new training set, in order not to let the resulting model forget what it has already previously learned. All these experiments have been conducted with different amounts of data, with the purpose of finding the lower bound to the number of new or replay images. The idea behind this is that on an edge machine with a limited computing power, if it is possible to limit the number of images that have to be stored in memory, the entire system becomes far more viable.