Linear regression can be applied to time series data to extract model parameters such as the effective force and friction constant matrices of the system. Even highly nonlinear systems can be analyzed by linear regression, if the total amount of data is broken up into shorter “time windows”, so that the dynamics is considered to be piece-wise linear. Traditionally, linear regression has been performed on the equation of motion itself (which approach we refer to as LRX). There has been surprisingly little published on the accuracy and reliability of LRX as applied to time series data. Here we show that linear regression can also be applied to the time correlation function of the dynamical observables (which approach we refer to as LRC), and that this approach is better justified within the context of statistical physics, namely, Zwanzig-Mori theory. We test LRC against LRX on a simple system of two damped harmonic oscillators driven by Gaussian random noise. We find that LRC allows one to improve the signal to noise ratio in a way that is not possible within LRX. Linear regression using time correlation functions (LRC) thus appears to be not only better justified theoretically, but it is more accurate and more versatile than LRX.