This letter considers minimizing the bit error rate (BER) of unmanned aerial vehicle (UAV) communications assisted by intelligent reflecting surfaces (IRSs). By noting that increasing the number of IRS elements in the presence of phase errors does not necessarily improve the BER, it is crucial to use only the elements that contribute to reducing the BER. Consequently, we propose an efficient algorithm to activate only the elements that improve the BER. The proposed algorithm has lower complexity and comparable BER to the optimum selection process, which is an NP-hard problem. The accuracy of the estimated phase is evaluated by deriving the probability distribution function (PDF) of the least-square (LS) channel estimator, and showing that the PDF can be closely approximated by the von Mises distribution at high signal-to-noise ratios (SNRs). The obtained analytical and simulation results show that using all the available reflectors can significantly deteriorate the BER, and thus, elements’ selection is necessary. In particular scenarios, using about 26% of the reflectors provides more than 10 fold BER reduction.