Smartphone Telemedicine: A Novel Workflow for Creating Prosthetic Sockets Using Semi-automated Photogrammetry

Recent trends in the field of prosthetic technology are shifting towards all-digital approaches. These digital technologies have the potential to augment the current models of clinical healthcare and reach more amputees than currently possible. Unfortunately, most of the equipment needed to design and build digitally shaped prostheses is expensive or requires a high level of expertise. Objective: This paper developed several methods to enable the digital fabrication of prostheses using only low cost tools that do not require extensive specialized training. Methods: The authors built a cell phone application in Android Studio that enables automated collection of image data which can be rendered into accurate 3D-models using photogrammetry on a remote server. They then developed a new protocol to shape these residual limb models into functional prosthetic sockets using the freely available software Autodesk Meshmixer. Results: Finally, the authors ran a case study where they rectified a prosthetic socket for an amputee and compared this model to a socket built by a clinician. Conclusion: This case study showed that these new methods can successfully create accurate and functional prosthetic sockets. Significance: By leveraging ubiquitous low cost smartphone technology, the authors believe that amputees in underserved areas, who currently face large barriers both in cost and physical access to proper care, could soon be able to obtain affordable prosthetic limbs.


I. INTRODUCTION
Major limb amputation is one of the largest issues our global community faces. Approximately 40 million individuals are amputees and an estimated 95% do not have access to a prosthetic limb [1]. Sadly, these amputees suffer from systemic disadvantages and their disability can limit access to fundamental opportunities needed for modern society [2]. Since the 1980s, prosthetic researchers have been interested in integrating computer technologies into the clinical practice for manufacturing prosthetic sockets. Digital technologies have the power to increase both the speed and ease of manufacturing prosthetic sockets while decreasing the cost to manufacture these devices. Despite over 30 years of development in computer aided design (CAD) and computer aided manufacturing (CAM), widespread implementation of these technologies are still limited [3] [4]. However, the benefits of these technologies are so significant that they undoubtedly will shape the future of prosthetic socket manufacturing. Specifically, virtual design workflows can decrease the need for in-person patient interactions during initial design phases and can increase the productivity of skilled prosthetists [5]. During these phases, a prosthetist will be able to digitally simulate different shape configurations to account for a patient's needs. While there is no substitution for direct contact, this capability extends the reach of a prosthetist and can allow them to service amputees across a greater area. In addition, CAM methods are more consistent than traditional manufacturing. The end product is less reliant on a prosthetist's experience ensuring that an amputee can get a higher quality prosthesis [6].
Digital workflows for creating prosthetic sockets have several key features in common. In general, these workflows encompass three stages: scanning, rectification, and manufacturing. During the scanning stage, a prosthetist aims to obtain a geometrically accurate model of an amputee's limb. In the rectification stage, this limb model is then modified and shaped digitally into a prosthetic socket while accounting for patient specific physiology. Finally, this digital socket model is brought into reality using a computer controlled manufacturing process. Two major challenges for these virtual workflows is that the technologies currently used for these stages are expensive and inaccessible. In this paper, we describe efforts to create low cost methods in two of the three key areas for automated manufacturing of prosthetic sockets: scanning and rectification. We leverage the ubiquity of android smartphones (with billions of users worldwide) to create an accessible platform to reach amputees around the world. Fig. 1 shows the author's cell phone application being implemented in a clinical setting.

II. TECHNICAL BACKGROUND
Rather than develop new 3D reconstruction algorithms, the commercial software Agisoft PhotoScan was utilized for model rendering. However, understanding photogrammetric reconstruction algorithms is critical to determine what data a cell phone application needs to collect automatically. In addition, understanding current clinical practice for designing prosthetic sockets is equally important to design a new virtual modeling workflow.

A. Background on Photogrammetry
Photogrammetry is defined as "science of measuring in photos". It provides a mathematical relationship between two dimensional photos and three dimensional objects. Fundamentally, for any object point that is represented in at least two photographs, it is possible to calculate where this point exists in a three dimensional coordinate system [7]. Dawson-Howe Vernon [8] developed a pinhole camera model for calculating the calibration matrices. Epipolar theory allows us to determine the baseline of multiple photos and their position [9]. All these theories and techniques give us the ability to 3D reconstruct objects using photographs with different angle views.
To successfully render a static 3D model from images, a sufficient quantity of photos with an overlap of between 80 and 90% between them is needed [10]. With this input data set, the 3D reconstruction algorithm will calibrate the camera, calculate the matched feature points in different cameras, and infer the relative camera poses in 3D space. Once calculation of camera poses is finished, the algorithm calculates the essential matrix and fundamental matrix for this data set [9]. Finally, the algorithm generates a dense point cloud of the object from the matched and calculated feature points; a 3D mesh can be easily rendered from this point cloud data [10]. With the correct inputs, photogrammetric 3D models generated from smartphones are capable of producing sub-millimeter accuracy and precision [11].

B. Background on Designing Prosthetic Sockets Digitally (Rectification)
Prosthetists and engineers have known for a long time what features rectification software needs to have. For instance, Bo Klasson et al. [12] specified that an ideal socket rectification software would be an "automated system" that could help "avoid duplication of work, simplify studying three-dimensional geometry-avoiding physical models, simplify input of data for analyses and display of results, simplify documentation of the product, and store experience and information from previous designs." However, even after more than 20 years of development, Rogers et al. [3] noted that software inadequacies for digital socket rectification was the largest hurdle for designing prosthetic sockets virtually. This is because development of socket rectification software has been more gradual and reflects closely advancements in computer aided design (CAD).
The socket modeling assistant (SMA) is a semiautomated socket rectification method developed by Colombo et al. [13] and Buzzi, et al. [14]. Key advances that this software implementation brings include importing parameters to make suggestions for socket rectification. However, SMA requires a high quality residual limb model typically generated from CT or MRI scans, which can be expensive and difficult to obtain. OMEGA is a commercial CAD software package developed by Ohio WillowWood that has been widely used in prosthetic clinics. It features "tools for shape capture, design and fabrication," that include shape alignment, landmark identification, and a goniometer tool for measuring and changing angles [15]. The software requires a high level of expertise in order to be used to its maximum potential. NiaFit is a socket rectification software built on the Autodesk Meshmixer platform, which enhances the functionality of digital socket rectification by creating a simplified workflow [16]. Unfortunately, this software is currently under development and is not available for widespread use.
Current frontiers in digital socket rectification utilize genetic algorithms and eigenvector algorithmic methods to create shaping approaches that are nearly fully automated [17] [18]. These new technologies hold great promise for fulfilling Bo Klasson et al. [12] original goals from 1985 for a fully automated rectification approach; unfortunately these technologies are far from being implemented commercially.

A. Development of an Android Application for Semiautomated Photogrammetric Reconstruction
Photogrammetry is a powerful tool for reconstructing accurate models of 3D objects from 2D photographs. It has been recently identified as a solution for building limb models of transtibial amputees [19]. It is highly accurate and can generate models at low cost, but currently is very Page 2 of 7   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59 time intensive and requires specialized knowledge. These barriers have led the technique to be underutilized in the field of prosthetics and orthotics. One challenge of utilizing photogrammetry is that photographs (inputs) need to be of sufficient quality and with a large degree of overlap in order to successfully render 3D models (outputs). Unfortunately, existing low-cost solutions for automation are unreliable and unrepeatable [11]. Being able to consistently and quickly collect this data was a major challenge, so we determined it was essential to build a new tool to help automate the process. We decided that utilizing a smartphone would be the best platform for collecting this data since smartphones contain high resolution cameras and a suite of embedded sensors that can be utilized jointly within an application. In addition, since this application needed to be as accessible as possible, we built this program within the Android development framework. Using this application, it is faster and simpler to capture accurate photogrammetry data to render 3D models of amputee's residual limbs  1) Implementation of Critical Functions: This cellphone application addresses several critical bottlenecks in the 3D reconstruction algorithm. Specifically, this application needed to collect photographic data in a complete 360 field around an object with at least 80% overlap. This application automatically begins a capture sequence at set angular intervals of 10 degrees (for a minimum of 36 captures per session). At a distance of 0.5 m, these photos should have a minimum 86.1% of overlap. The application constantly queries the smartphone accelerometer and magnetometer to calculate global orientation and update its state. During each capture sequence, the application takes a three photo burst. This photo burst ensures that there was at least one photograph in focus at each angular interval and also increases the degree of overlap up to 95.4%. Implementation of this function required our Android smartphones to have at least Android Version 5.0 in order to utilize Camera2 API. Barbero-Garcia et al. [11] found that spherical objects could be successfully rendered with as few as 26 images. The application collects a minimum data set of 108 photos which is sufficient for rendering models. As a safety check, the application has a "gallery" function so that users can examine the photographs and determine their quality. The users can then send this dataset to a remote cloud based server for post-processing. During post-processing, the image backgrounds are removed to improve the chance of a successful reconstruction.
2) Application Benchmark: Previous work by Barbero-Garcia et al. [11] showed that smartphone photogrammetric methods can achieve sub-millimeter accuracy and precision under ideal conditions. However, some of these conditions are not possible outside of a laboratory setting. We tested the performance of our application in a challenging real world environment. We scanned a residual limb model using the application in partially shaded outdoor lighting. This limb model did not have any markers to aid in tie point detection. Despite these challenging input conditions, the models retained sub-millimeter accuracy on average (I). However, the local precision suffered and its average on these models was limited to 3.6 mm. This can be clearly observed as a rough surface texture visible in Fig. 3. We can account for the loss of local precision with mesh smoothing functions during the rectification step. Since the accuracy was high in these renderings, smoothing the surface would increase the precision of the limb models. Since they do not need this degree of accuracy [4], the limb models rendered from data collected with our application are sufficient to be rectified into prosthetic sockets.

B. Development of Virtual Socket Modeling Workflow
Socket rectification (i.e. shaping a limb model into a prosthetic socket) is the most critical step in any virtual model workflow for designing prosthetics. In this step, prosthetists make the most important design decisions that will affect an amputee's clinical outcomes. Based on limitations in previous work, there was a need to create a method for socket rectification that utilized only free open source tools. The following workflow is important because having a freely accessible rectification protocol would enable others to completely construct a prosthetic socket digitally and at a low/no cost. This rectification workflow can be paired with the aforementioned mobile photogrammetry application for a completely low cost modeling environment. Page 3 of 7   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58 59 60 Because of the limitations in some of the aforementioned socket rectification softwares and because there was a need to create a free and open access tool for designing prosthetic sockets, we developed a protocol for shaping transtibial prosthetic sockets in AutoDesk Meshmixer.
To begin generating a socket model, a geometry file containing the residual limb shape is imported into Autodesk Meshmixer. After importing the residual limb model to Meshmixer, surface imperfections are removed from the model using the smooth tool. However, it should be noted that smoothing the residual limb model may change some of its geometry. Some important features may be lost, and due to volume loss on the residual limb model, the "pistoning" effect might become more pronounced and decrease comfort [20]. Thus this volume change should be accounted for when creating the shell for the prosthetic socket.
Once the limb model has been satisfactorily smoothed, this model is similar to the positive plaster casts traditionally created by prosthetists. To begin reshaping the model, it is important to understand the areas of the residual limb that are tolerant and intolerant to pressure. Some pressure tolerant areas, known as load zones, in a transtibial amputee are the patellar ligament, medial tibial flare, anterior compartment, medial shaft of tibia, lateral shaft of fibula, and the posterior compartment [14]. In general, these are the areas without bony protuberances or tendons. Prosthetists typically remove material from these load zones when creating the positive plaster cast. The off load zones, or pressure intolerant areas, include the patella, lateral tibial flare, anterior tibial flare, anterior tibial tubercle, crest of tibia, distal end of tibia, head of fibula and peroneal nerve, distal end of fibula, and the hamstrings tendons. These areas are typically where bony protuberances or tendons are located. Material is added in the positive plaster cast in order to relieve pressure from these areas. Fig. 4 shows a diagram of these areas. Patellar-tendon bearing (PTB) sockets are the most common design for transtibial sockets. The PTB socket applies load primarily to the patellar ligament, while also relying on other pressure-tolerant areas of the residual limb. This traditional construction method is thoroughly outlined by Fleer and Wilson [21]. This protocol addressed the pressure tolerant areas by using the shrink smooth tool in Meshmixer to remove material from the residual limb. Based on recommendations by [21], 3.17 mm was removed from the anteromedial tibial flare to improve anteromedial stability. Similarly, 3.17 mm was removed from the posterior compartment to provide counter support against anterior forces and improve stability. Lastly, material in the anteromedial and anterolateral surfaces are also removed, creating a wedge-like, triangular shape that improves comfort and anteroposterior stability. To create the patellar tendon indent, a cylinder was imported from Meshmixer's template files, resized and aligned with the patellar tendon, and performed a Boolean difference. This indentation was 12.7 mm deep, 12.7 mm long, with a radius of about 25.4 mm. Following these adjustments, material was added to pressure intolerant zones to reduce contact between the socket and the residual limb in these areas. The size of the offsets were typically 25.4 mm. It is also important to keep weight entirely off the sensitive distal end of the residual limb, so the residual limb is extended distally by at least 50 mm, depending on the height of the amputee Fig. 5. After the pressure zones were addressed, condylar flares were incorporated into the design to "add contact and support on the medial and lateral aspects of the distal femur" [22]. Condylar flares also help with suspension and reduce Page 4 of 7 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59 pistoning. These flares, as well as the rear lip, were created by importing a cube that is slightly wider than the residual limb, combining the two objects through a Boolean addition, and smoothing the surface to create a gentle curvature. The socket boundary is then outlined using the select tool, and the socket shell is created using the offset tool, with the offset amount accounting for the change in volume due to smoothing and the presence of a liner. According to Buzzi, et al. [14], the suggested thickness of the socket in mm would be the patient's weight in kilograms divided by 20; however, the final thickness depends on the materials and manufacturing process used.
Additional steps are required to prepare the socket for attachments and suspension systems but these do not largely interfere with the primary rectification process. A video demonstrating the Meshmixer protocol is available at: https://youtu.be/zoqGtojqeyE

IV. VALIDATION OF NEW METHODS: CASE STUDY ON AMPUTEE
Once the low cost methods for scanning and rectification were developed, a case study was ran to see how these technological advances would perform in real world conditions. In the following case study, we test the entire novel digital workflow for building a prosthetic socket, from scanning, to rectification, and finally manufacturing. We then evaluate how this prosthetic socket compares to a design made using traditional clinical practice to understand the performance of our new workflow.

A. Fabrication of Test Socket for Amputee
Using the methods described earlier, a prosthetic socket was designed and fabricated for a unilateral transtibial amputee in accordance with the UCSD Human Research Protections Program (male, age: 39 years, height: 177 cm, weight: 68 kg, activity level: K4, cause of amputation: trauma, time since amputation: 4 years). Fig. 6 shows the process workflow, going from limb photo capture and 3D rendering, to socket rectification, and finally, printing the socket and testing it on the amputee. We captured the necessary photographic data using our cell phone application in shaded outdoor lighting conditions. We then used Agisoft Metashape to render the residual limb model. Following this, we then imported the limb model into Autodesk Meshmixer and rectified the socket. Once we created a satisfactory socket design, we manufactured a test socket using a Raise3D Pro2Plus printer. A deviation from the shaping method outlined above is that we integrated a flat, circular bottom on the prosthetic socket for easy pylon attachment. Further, we drilled a small hole into the posterior, distal end of the socket to allow air to escape when donning. A prosthetic sleeve was primarily used as the suspension system. The socket was attached to the pylon using a standard pyramidal adapter. It took 4.5 hours to complete the entire workflow, and an additional 13 hours to manufacture. The overall cost of the final socket was $50, using 765 grams of PLA filament for the socket, and the additional components.

B. Performance of New Workflow in Case Study
One major question that this case study sought to address was: how did a socket generated with this workflow compare to one made by traditional methods in a prosthetic clinic? We collaborated with a local prosthetic clinic to obtain a digital socket model for the amputee case subject. This digital socket model was compared with the rectified socket model using CloudCompare. The geometric tolerances were close between the model built by the prosthetic clinic and the one generated by the new workflow with an average distance of 1.414 mm and local precision (standard deviation) of 3.201 mm Fig. 7. Most critically, the major geometrical differences between the two models were due to specific design choices for clinical outcomes. Thus both models are suitable for the amputee to utilize (the amputee was able to  comfortably wear both prosthetic sockets.) As evident in Fig.  8, the largest difference between the socket built with the new workflow and by the prosthetic clinic is that the clinician added extra allowance for the distal end of the amputee's limb. In addition, the rectified model added pressure zones to the medial and lateral sides of the tibia below the patellar tendon to a much larger extent than the model from the prosthetic clinic. The amputee was able to clearly distinguish both design choices, and each had strengths and weaknesses for comfort and fit.

V. CONCLUSIONS AND FUTURE RESEARCH
Overall, this novel workflow marks a significant milestone in the development of prosthetic sockets digitally and automatically at a low cost. The cellphone application we developed greatly reduces gaps in models rendered with photogrammetry and improves the ability to generate 3D models consistently. The prosthetic socket rectification workflow we developed can successfully render models at low cost using only freely available tools. In the case study presented herein, the prosthetic socket model generated with the novel workflow was on average within 1.4 mm of a socket model built with current clinical methods. This means that the workflow can create socket designs accurate enough to be manufactured and worn.
These tools had certain limitations that inspire our future research. Although the cellphone application reduced the burden of image acquisition for photogrammetry, these images had to be post processed manually. We could achieve a greater level of automation by creating background removal tools utilizing computer vision techniques. In addition, while the rectification method was based on existing clinical practice, it would be preferable to develop a fully automated shaping method based on historically successful socket models. This could be achieved with a machine learning algorithm that could transform point cloud data based on previous models. However, the dataset to train a machine learning model such as this does not currently exist. Collecting this data would be an exciting first step towards fully automated socket rectification.
The authors foresee that by developing this low cost mobile platform, many amputees who faced accessibility barriers will now have the opportunity to obtain an inexpensive prosthetic limb. This platform can be scaled and ultimately will help clinicians to build larger networks to interact with patients in a broad geographic zone. Since all of the socket models can be built remotely, clinicians can collaborate with a much larger global additive manufacturing infrastructure to build these prostheses affordably. Since the mobile platform is such a flexible tool, it could also be utilized to scan and manufacture many other replacement components on demand. This technology is one step towards paving a brighter future for amputees around the world.

ACKNOWLEDGMENT
Funding for this research was provided by the Benbough Foundation, Alfred P. Sloan Foundation, UC San Diego Graduate Division, UCSD Center for Human Frontiers, and the Qualcomm Institute at UCSD. The authors would like to give a special thanks to the generous support from the Center for Human Frontiers and the Qualcomm Institute at UC San Diego for providing resources for this research project. The authors would like to extend their gratitude to Peter Harsch Prosthetics in San Diego for helping generate limb models and sockets for the case study. In addition, the authors appreciate Bhagwan Mahaveer Viklang Sahayata Samiti (Jaipur Foot) for generously sharing their knowledge of prosthetic manufacturing technology. The authors would like to thank Timothy Ho, Abraham Jarque, Kevin Dang, Morgan Lyew, James Strawson, Luca DeVivo Nicoloso, and Professor Falko Kuester for providing the inspiration to pursue this research project. Finally, the authors would like to thank all of the other students involved on the Project Lim(b)itless team for their incredible support and positive energy which led to the successful completion of this research. Page 6 of 7   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60