DFT21: Discrete Fourier Transform in the 21st century
- Prasanta Pal ,
- Shataneek Banerjee ,
- Amardip Ghosh ,
- David R. Vago ,
- Judson Brewer
Abstract
Knowingly or unknowingly, digital-data is an integral part of our
day-to-day lives. Realistically, there is probably not a single day when
we do not encounter some form of digital-data. Typically, data
originates from diverse sources in various formats out of which
time-series is a special kind of data that captures the information
about the time-evolution of a system under observation. How- ever,
capturing the temporal-information in the context of data-analysis is a
highly non-trivial challenge. Discrete Fourier-Transform is one of the
most widely used methods that capture the very essence of time-series
data. While this nearly 200-year-old mathematical transform, survived
the test of time, however, the nature of real-world data sources
violates some of the intrinsic properties presumed to be present to be
able to be processed by DFT. Adhoc noise and outliers fundamentally
alter the true signature of the frequency domain behavior of the signal
of interest and as a result, the frequency-domain representation gets
corrupted as well. We demonstrate that the application of traditional
digital filters as is, may not often reveal an accurate description of
the pristine time-series characteristics of the system under study. In
this work, we analyze the issues of DFT with real-world data as well as
propose a method to address it by taking advantage of insights from
modern data-science techniques and particularly our previous work SOCKS.
Our results reveal that a dramatic, never-before-seen improvement is
possible by re-imagining DFT in the context of real-world data with
appropriate curation protocols. We argue that our proposed
transformation DFT21 would revolutionize the digital world in terms of
accuracy, reliability, and information retrievability from raw-data.