Rare event detection (RED) involves the identification and detection of events characterized by low frequency of occurrences, but of high importance or impact. This paper presents a Systematic Review (SR) of rare event detection across various modalities using Machine Learning (ML) and Deep Learning (DL) techniques. This review comprehensively outlines techniques and methods best suited for rare event detection across various modalities, while also highlighting future research prospects. To the extent of our knowledge, this paper is a pioneering SR dedicated to exploring this specific research domain. This SR identifies the employed methods and techniques, the datasets utilized, and the effectiveness of these methods in detecting rare events. Four modalities concerning RED are reviewed in this SR: video, sound, image, and time series. The corresponding performances for the different ML and DL techniques for RED are discussed comprehensively, together with the associated RED challenges and limitations as well as the directions for future research are highlighted. This SR aims to offer a comprehensive overview of the existing methods in RED, serving as a valuable resource for researchers and practitioners working in the respective field.