The localization performance of multi-state constraint Kalman filter (MSCKF)-based visual-inertial odometry (VIO) is suffering from linearization errors on the feature three-dimensional (3D) positions and delayed measurement updates. Targeting more accurate and robust localization capability, we incorporate the pose-only representation into the filter-based VIO and propose a pose-only representation-based Kalman filter (PO-KF) in this paper. Leveraging the decoupling of camera poses and feature positions in the pose-only representation, the proposed PO-KF explicitly eliminates feature 3D coordinates from its measurement equation, efficiently removing the linearization errors caused by feature positions and ensuring immediate updates of visual measurements. In the proposed PO-KF, we also introduce an information matrix-derived base-frame selection algorithm for the pose-only representation, which identifies the most suitable base-frames for each feature. Extensive experiments on multiple datasets demonstrate the dramatically improved localization accuracy and robustness of PO-KF compared to MSCKF, along with comparable real-time performance.