This paper deals with a new spatial domain-based self-interference cancellation (SIC) method called constrained minimum mean square error (C-MMSE) for massive multiple-input multiple-output (mMIMO) full-duplex (FD) communication systems. The main idea is to treat the self-interference (SI) signal emitted from an FD node as a particular spatial stream arriving at the receiver part of that same FD node which needs to be spatially postcoded along with other useful signals coming from other transmitters, so that it falls into the null space of the MIMO channel that includes the FD node transmitter part as an input. On this basis, we first adapt the expressions of the spatial combiners with respect to the conventional zero forcing (ZF) and minimum mean square error combining (MMSE) criteria and show that the latter is not capable to efficiently cancel the SI signal unless an additional constraint is added to properly perform SIC. We hence design the new so-called C-MMSE combiner and derive its expression. In addition to our proposal, the originality of our work lies in the consideration of spherical wave model (SWM) for modeling the SI channel, which is justified by the close proximity of the transmit and receive antenna panels in the FD node.

We examine and compare the SIC performance of the adapted ZF combiner, the adapted MMSE combiner and the newly introduced C-MMSE combiner by evaluating the obtained spectral efficiency (SE). We also highlight the robustness of the SWM-based SI channel modelling compared to conventional planar wave modelling (PWM) and therefore the relevance of using it.