Abstract

Assisted and automated driving functions will rely on machine learning algorithms, given their ability to cope with real world variations, e.g. vehicles of different shapes, positions, colours, etc. Supervised learning needs annotated datasets, and several datasets are available for developing automotive functions. However, these datasets are tremendous in volume, and labelling accuracy and quality can vary across different datasets and within dataset frames. Accurate and appropriate ground truth is especially important for automotive, as "incomplete" or "incorrect" learning can impact negatively vehicle safety when these neural networks are deployed. This work investigates ground truth quality of widely adopted automotive datasets, including a detailed analysis of KITTI MoSeg. According to the identified and classified errors in the annotations of different automotive datasets, this paper provides three different criteria collections for producing improved annotations. These criteria are enforceable and applicable to a wide variety of datasets. The three annotations sets are created to: (i) remove dubious cases; (ii) annotate to the best of human visual system; (iii) remove clear erroneous bounding boxes. KITTI MoSeg has been reannotated three times according to the specified criteria, and three state-of-the-art deep neural network object detectors are used to evaluate them. The results clearly show that network performance is affected by ground truth variations, and removing clear errors is beneficial for predicting real world objects only for some networks. The relabelled datasets still present some cases with "arbitrary"/"controversial" annotations, and therefore this work concludes with some guidelines related to dataset annotation, metadata/sub-labels, and specific automotive use-cases.