Using computer simulations, we analyze the bit-error rate (BER) characteristics of the optical transmission system repeated by erbium-doped fiber amplifiers (EDFAs) and two-mode phase-sensitive optical parametric amplifiers (PSAs), where  phase-conjugated (PC) twin waves are co-propagating through the link. We focus on the 256 quadrature-amplitude-modulation (QAM) format, whose transmission characteristics are seriously affected by fiber nonlinearity because the required power is significantly high.

Changing the repeater spacing and group-velocity dispersion (GVD) of fibers for transmission in the two-mode PSA-based system, we calculate the BERs as a function of the launched power. When the repeater spacing and  GVD value of fibers are properly designed, the two-mode PSA-based system can mitigate nonlinear impairments more effectively than the EDFA-based PC twin system.