Broadband reliable communication is a competitive 5G technology for cognitive communication scenarios, but meanwhile introduces multiform interference to existing broadband transform domain communication system (TDCS) transmission. In order to facilitate the improvement of the anti-jamming performance for the coexistence of diverse interference and TDCS signals in wireless heterogeneous networks, it is important to separated and eliminate various interference to TDCS systems. In this paper, a novel sparse learning method-based cognitive transformation framework of interference separation is formulated for accurate interference recovery, which can be efficiently solved by iteratively learning the prior sparse probability distribution of the interference support. To further improve the separation accuracy and iterative convergence, the principal component analysis and Bayesian perspective in orthogonal base learning are exploited to singly recover the multiple interference and TDCS signals. Moreover, utilizing different sparsity states of spectrum analysis, the proposed novel interference separation algorithm is extended to simultaneous separation based on state evolving of approximation message passing, which iteratively learns the belief propagation posteriors and keeps shrunk by iterative shrinkage threshold. Simulation results demonstrate that the proposed methods are effective in separating and recovering the sparse diversities of interference to TDCS systems, and significantly outperform the state-of-the-art methods.