Force/torque sensing on hand-held tools enables control of applied forces, which is often essential in both tele-robotics and remote guidance of people. However, existing force sensors are either bulky, complex, or have insufficient load rating. This paper presents a novel force sensing modality based on differential magnetic field readings in a collection of sensor modules placed around a tool or device. The instrumentation is easy to install and low profile, but nonetheless achieves good performance. A detailed mathematical model and optimization-based design procedure are also introduced. The modeling, simulation, and optimization of the force sensor are described and then used in the electrical and mechanical design and integration of the sensor into an ultrasound probe. Through a neural network-based nonlinear calibration, the sensor achieves average root-mean-square test errors of 0.41 N and 0.027 Nm compared to an off-the-shelf ATI Nano25 sensor, which are 0.80% and 1.16% of the full-scale range respectively. The sensor has an average noise power spectral density of less than 0.0001 N/sqrt(Hz), and a 95% confidence interval resolution of 0.063 N and 0.0086 Nm. The practical readout rate is 1.3 kHz over USB serial and it can also operate over Bluetooth or Wi-Fi. This sensor can enable instrumentation of manual tools to improve the performance and transparency of teleoperated or autonomous systems.

For many augmented reality guidance, teleoperation, or human-robot interaction systems, accurate, fast, and robust 6 degree of freedom object pose tracking is essential. However, current solutions easily lose tracking when line-of-sight to markers is lost. In this paper we present a tracking system which matches or improves on current methods in speed and accuracy, achieving 1.77 mm and 1.51 degrees accuracy at 22 Hz. Reflective markers are segmented in infrared images using contour detection before using the known marker geometry to perform point correspondence and pose computation using novel approaches. At the same time, a new square root unscented Kalman filter is introduced which improves accuracy and flexibility by tracking the markers themselves rather than the computed pose, and enables fusion of an external inertial measurement unit. This reduces noise and makes the tracking robust to brief loss of line-of-sight. The algorithms and methods are described in detail with pseudo-code for ease of reproduction. The system is implemented in simulation and on a Microsoft HoloLens 2 using Unity for ease of entry and integration into graphical projects. The code is made available open source. Tests of the system are described, and the results analyzed .

We previously introduced a novel mixed reality (MR) teleguidance system, human teleoperation [1,2], in which a human (expert) leader and a human (novice) follower are tightly coupled through MR and haptics for applications such as tele-ultrasound. In this paper, a communication system suitable for human teleoperation is presented and characterized in various network conditions, over Ethernet, Wi-Fi, 4G LTE, and 5G. To study all types of latency in the system, the human response time is additionally characterized through step response tests with 11 volunteers. The step responses were obtained by tracking the position and force of the human hand in response to a change in the MR target. The round-trip communication latency is 40+/-10 ms over 5G, and down to 1+/-0.6 ms over Ethernet for typical throughputs. The human response time to a step change in position depends on the step magnitude, but is 485-535 ms, while the reaction time for forces is 150-200 ms. Both lags are decreased when tracking smooth motions. Thus, we demonstrate that the system is network agnostic and can achieve good teleoperation performance and secure, fast communication in appropriate network conditions. The presented tools and concepts are applicable to any high-performance teleoperation system, for example for remote surgery.

Purpose: In “human teleoperation” [1], augmented reality (AR) and haptics are used to tightly couple an expert leader to a human follower. To determine the feasibility of human teleoperation, we quantify the abil- ity of humans to track a position and/or force trajectory via AR cues. The human response time, precision, frequency response, and step response were characterized, and several rendering methods were compared. Methods: Volunteers (n=11) performed a series of tasks as the fol- lower in our human teleoperation system. The tasks involved tracking pre-recorded series of motions and forces, each time with a different rendering method. The order of tasks and rendering methods was ran- domized to avoid learning effects and bias. The volunteers then performed a series of frequency response tests and filled out a questionnaire. Results: Rendering the full ultrasound probe as a position target with an error bar displaying force led to the best position and force tracking. Following force and pose simultaneously was more difficult but did not lead to significant performance degradation versus following one at a time. On average, subjects tracked positions, orientations, and forces with rms tracking errors of 6.2 ± 1.9 mm, 5.9 ± 1.9˚, 1.0 ± 0.3 N, steady-state errors of 2.8 ± 2.1 mm, 0.26 ± 0.2 N, and lags of 345.5 ± 87.6 ms respectively. Performance decreased with input frequency, until the person could no longer follow, depending on the input amplitude. Conclusion: This paper characterizes human tracking ability in aug- mented reality human teleoperation, which shows the system’s feasi- bility and good performance, and is important for designing future human computer interfaces using augmented and virtual reality.

Current teleguidance methods include verbal guidance and robotic teleoperation, which present tradeoffs between precision and latency versus flexibility and cost. We present a novel concept of “human teleoperation” which bridges the gap between these two methods. A prototype teleultrasound system was implemented which shows the concept’s efficacy. An expert remotely “teloperates” a person (the follower) wearing a mixed reality headset by controlling a virtual ultrasound probe projected into the person’s scene. The follower matches the pose and force of the virtual device with a real probe. The pose, force, video, ultrasound images, and 3-dimensional mesh of the scene are fed back to the expert. In this control framework, the input and the actuation are carried out by people, but with near robot-like latency and precision. This allows teleguidance that is more precise and fast than verbal guidance, yet more flexible and inexpensive than robotic teleoperation. The system was subjected to tests that show its effectiveness, including mean teleoperation latencies of 0.27 seconds and errors of 7 mm and 6◦ in pose tracking. The system was also tested with an expert ultrasonographer and four patients and was found to improve the precision and speed of two teleultrasound procedures.

Purpose Respiratory motion during positron emission tomography (PET) scans can be a major detriment to image quality in oncological imaging, leading to loss of quantification accuracy and false negative findings. The impact of motion on lesion quantification and detectability can be assessed using anthropomorphic phantoms with realistic anatomy representation and motion modelling. In this work we design and build such a phantom, with careful consideration of system requirements and detailed force analysis. Methods: We start from a previously-developed anatomically-accurate shell of a human torso and add elastic lungs with a highly controllable actuation mechanism which replicates the physics of breathing. The space outside the lungs is filled with a radioactive water solution. To maintain anatomical accuracy in the torso and realistic gamma ray attenuation, all motion mechanisms and actuators are positioned outside of the phantom compartment. The actuation mechanism can produce a plethora of custom respiratory waveforms with breathing rates up to 25 breaths per minute and tidal volumes up to 1200mL. Results: Several tests were performed to validate the performance of the phantom assembly, in which the phantom was filled with water and given respiratory waveforms to execute. All parts demonstrated nominal performance. Force requirements were not exceeded and no leaks were detected, although continued use of the phantom is required to evaluate wear. The respiratory motion was determined to be within a reasonable realistic range. Conclusions: The full mechanical design is described in this paper, as well as a software application with graphical user interface which was developed to plan and visualize respiratory patterns. Both are available open source and linked in this paper. The developed phantom will facilitate future work in evaluating the impact of respiratory motion on lesion quantification and detectability.