FrequentNet: A frequency-based neural network architecture with joint temporal and frequency domains

The frequency domain plays a crucial role in image processing. However, modern neural networks, such as Convolution Neural Networks and Transformers, only operate in the temporal domain, resulting in a contradiction concerning information aggregation. In contrast, the frequency domain has distinct advantages to solving the contradiction. In this paper, we introduce a frequency-based neural network architecture with joint temporal and frequency domains named as FrequentNet. We analyze the challenges in frequency-based neural networks associated with combining temporal and frequency domain information. Moreover, we find that the absence of frequency-domain downsampling methods and complex computations also affect the frequency models' performance. To tackle the abovementioned problems, we introduce a residual connection that separates the temporal and frequency domains to resolve information aliasing. Furthermore, we devise a frequency domain down-sampling method based on the mapping. Finally, we use Discrete Cosine Transform as the frequency domain transformation operator to avoid the need for complex computations.  Comprehensive experiments demonstrate that our approach surpasses existing frequency-based backbones in diverse fields, including image classification, object detection, and semantic segmentation, whose superiority stems from the frequency domain's robust and efficient information aggregation capability.