This paper presents a novel feature-line points tracking algorithm designed to be highly efficient and accurate for real-time monocular line-based SfM applications in man-made environments. The proposed algorithm exploits feature-line points proprieties to detect and extract long line segments and ensure their temporal evolution in the subsequent frames while handling line segment detection and tracking issues such as noisy images, over-segmentation, edge thresholding, false detections, occlusions, and ambiguous matches. To fulfill the real-time and embeddability constraints of our proposed method, we adopt the hardware/software co-design to achieve a suitable implementation with a scalable FPGA-based embedded heterogeneous architecture. Based on this, experimental results show the portability of the line segment detector block of this proposed algorithm on scalable FPGA-based embedded heterogeneous architectures, improving the acceleration of the sequential execution by about 98\%, whereas a 53\% improvement with GPU-based heterogeneous architectures. For qualitative and quantitative assessment, we exhibit extensive benchmarking with other state-of-the-art algorithms regarding feature-line segment extraction and tracking, showing the efficiency of the proposed algorithm and ensuring very hopeful real-time performance using two common datasets.