Reconfigurable intelligent surface (RIS) is an emerging technology for improving the performance of wireless communications. In this letter, a joint Squared-Sine function and atomic norm minimization (ANM), denoted as SS-ANM, is proposed for passive sensing assisted by RIS in wireless communication systems interfered by a wireless access point (AP). The proposed joint optimization is used to estimate the targets’ direction-of-arrival (DOA), employing the spatial domain sparsity of the received signals. The optimal solution is achieved from the alternating direction method of multipliers (ADMM). With the help of the Squared-Sine function as the cost function, the proposed solution provides better estimation accuracy under various noises besides Gaussian noise, and meanwhile requires a lower computation complexity. Simulation results verify the superiority of the proposed joint minimization over existing DOA estimation methods.