Specular Reflection Detection and Removal Based on Deep Neural Network for Endoscope Images

— Specular reflections have always been undesirable when processing endoscope vision for clinical purpose. Scene afflicted with strong specular reflection could result in visual confusion for the operation of surgical robot. In this paper, we propose a novel model based on deep learning framework, known as Surgical Fix Deep Neural Network (SFDNN). This model can effectively detect and fix the reflection points in different surgical videos hence opening up a whole new approach in handling undesirable specular reflections. Clinical Relevance — This method significantly alleviates the specular reflections caused by intense lighting condition during minimally invasive surgeries, which use illuminated endoscope.


I. INTRODUCTION
Currently, the number of cancer cases in China is the highest in the world. The demand for accurate and efficient surgical robot in China grows continuously as results. However, most of the robot-based surgical operations through endoscope vision suffered from specular reflections caused by metallic instruments or tissue membrane. These reflections obscure the features of critical organs and tissue, causing much visual confusion ( Figure 1).

Figure 1: Existing Reflection Points in One Video
Many approaches have been proposed to detect and restore affected areas affected by specular reflection. Some proposed approaches concentrate on the processing of fixed images [1] [2] [3]. However, they are not able to effectively process surgical videos. Some approaches took advantage of methods like KNN, Decision Tree and SVM to detect the specular reflection. However, these approaches are likely to overestimate the reflection area leading to erroneous judgment.
In this paper, we present a novel model based on deep learning framework, known as Surgical Fix Deep Neural Network (SFDNN). This model basically consists of two parts. In the first part, we designed and modified a deep neural network model to detect the reflection points in a surgical video. In the second part, we used the color mapping method to repair the reflection part, restoring the real surgical scene.

II. METHODS
By taking advantage of the power of deep neural network [4], we first generate a classification model for the whole surgical video using some spatial points in a few screenshots, and we label the points individually in reflective regions and in non-reflective regions based on their color differences. After building the training set, our model will be trained under this specific set to learn the weights and bias , hence, yielding the probability for target points being classified into two different regions.
Subsequently, we deploy our established model on the whole surgical video with the speed of 30 fps, filling the reflective and non-reflective regions respectively with white and black pixels, depending on the difference of classification possibility for each point (Figure 2). Secondly, we design a color-mapping approach for fixing the reflection points in the surgical video. Due to the fixed perspective of the operation video, we restore the information for the reflective region of current image by doing identity mapping from other unaffected frames, hence, reducing the information loss to the greatest extent ( Figure 3).

Specular Reflection Detection and Removal Based on Deep Neural Network for Endoscope Images
Tianyu Liu 1 , Junyu Chang 1 , Chongyu Wang 1 , Liangjing Yang 1 , ZJU-UIUC Institute For the fixing approach, we also design an algorithm and flowchart, which is shown in Figure 4. The parameter 1807 means we have 1807 images in total. In addition, we named each image as state to avoid perplexity.

Figure 4: Workflow for Specular Points Repair
Finally, we utilize filters to smooth the image and eliminate the noise caused by textual mapping. We essentially adopt a framework based on prior information filling and posterior information repair to generate images without reflective regions.

III. RESULTS
We test four classification models based on machine learning theory for the reflective region recognition. The run time of deep neural network (DNN) model is shortest for prediction and total time (including time of training and prediction) according to Figure 4. In addition, the accuracy of DNN model is the best. We finally select the DNN model for subsequent processing.
We use the time series scheme to restore the reflection area, and the accuracy of it is close to 100%, which will be shown in the poster. Moreover, we also generate a color map based on collecting all useful posterior information. In this color map ( Figure 5), most of the specular points have been fixed except areas hidden by the robotic arm of a surgical robot. In this paper, we create a new reflective region recognition algorithm based on deep neural network, which can effectively identify the reflective region, especially in surgical video. For the surgical video, we design a new scheme based on the prior information filling to repair the identified reflective areas in the surgical video and achieve interesting results. In the future, we plan to do research related to reducing the effect brought by the robot's arm to generate one perfect color map.