In this article, we propose a novel deep learning framework to identify location-specific inaccurate segmentation for medical images to aid downstream analyses. The proposed method relies directly on the extracted segmentation related features and does not need to use the independent standard during the inference phase to identify erroneous regions in the computed segmentation. This is significant because image segmentation plays an indispensable role for automated quantitative analysis in modern precision medicine and the absence of case-by-case ground truth for medical images limits the segmentation quality assessment. Moreover, in contrast to our approach, to the best of our knowledge, current deep learning-based segmentation quality assessment models evaluate the segmentation quality globally and may ignore significant local segmentation errors in locations where diseases are prevalent. For the purpose of disease diagnosis and/or prognosis, it is critical to identify inaccurate segmentation in local regions. We used two publicly available datasets to evaluate our method and show that our method is robust and works well to localize small but significant inaccurate segmentation in the cases of both good and bad overall segmentation results for different imaging modalities.