Abstract

Although research on self-driving vehicles (SDV) started long ago, developing a robust perception system is still challenging for researchers in academia and industry. Object classification, object detection and tracking, road detection, semantic segmentation, and instance segmentation are all introduced as the perception problems of the autonomous driving platform. Tons of methods have already been proposed to solve these perception problems. Some methods use a single modality (image only or Lidar only), and others use multi-modality. This review paper categorizes all semantic segmentation methods used for self-driving vehicles into three categories depending on their modality. We compare these methods according to their performance and mention the datasets used for evaluation. Some of the most common datasets in this field are also discussed in this paper. We highlight the challenges and scope of future research and discuss the background of this research field so that this paper can be used as a base for a new researcher to start research on semantic segmentation. Impact Statement-Semantic segmentation is a very useful perception tool. It can be used to get per-pixel semantic labels of the scenarios captured by the sensors of autonomous vehicles, which can then be used to identify potential obstacles, based on which planning and control decisions are made. Although many works have discussed semantic segmentation models, there are not many review papers that consolidate these works from a sensory perspective. In this paper, we analyze single and multimodal model architectures and compare their performances to guide researchers in deciding which direction (Lidar or camera, Lidar and camera) may be beneficial for them. The most frequently used datasets in this arena and the main features of the data are all elaborately discussed. Furthermore, the challenges that researchers may encounter, and the scope of future research are also included. This paper could provide a one-stop source for new researchers working in this area.