Aestheticization of Ultrasound Images

,


I. Introduction
Ultrasonography (USG) or ultrasound-imaging [1], [2] is a universally adopted imaging technique that uses highfrequency sound waves to view inside parts of human body [3].A plethora of applications are facilitated by this imaging modality.It helps diagnose anatomic structure, cardiac motion as well as the causes of pain, swelling and infection in the body's internal organs.Additionally, it is also used to examine fetal development for continuous evaluation and monitoring [4], [5] of various developmental conditions.Several factors contribute to the popularity of this image technique including but not limited to lowcosts, portability, non-invasive nature and negligible sideeffects.Out of all possible application scenarios, a very special case arises when USG is utilized to image the growth of an unborn child to monitor fetal development.The very first ultrasound image [6], [7] of an unborn child is a piece of precious gift.It not only carries a plethora of clinically significant markers, but also it also serves as one of the most significant image of one's parenthood as well as for the child to get a surprise by looking back.Although, colorization of digital-images [8], [9] has been in practice for a long time [10], but it is critically important for medical images to be color-schematized without information-loss for enhanced visual clarity as well as enhancement of aesthetics.Adding clarity and aesthetics to fetal-USG imaging without information-loss to augment naturally grainy images is the main theme of this work.In about 14 weeks, the physical characteristics of the fetas should be identifiable [11] through USG.However, it is often the case that, the image is so noisy [12], [13], the subtle anatomic features of the baby are not clearly discernible.Also, the raw-image is often completely devoid of any kind of aesthetics whatsoever.On the other hand, maternal bonding is a very important emotional aspect of fetal ultrasound imaging [11], [14], [15], particularly the very first few ones, where these images often play the role of being the only visual proof of the existence of the fetus.Beyond everything, it plays a very significant role of being an emotional connector for parents to be able observe the signature of something very precious to their lives.We argue that, any effort to enhance the clarity and aesthetics of this imaging modality would solve a problem with great importance.

II. Method
Data orthogonalization is the fundamental principle behind the process of separating context specific elements within a given set of data [16].Independent component analysis attempts to address similar kinds of of problems.In the context of our image aestheticization methods the basic ideas is to separate various information scales broadly described in ref. [17] although we use only the parts relevant to image-processing with suitable set of parameters (e.g.small kernel-size, and moderate threshold value).

A. Statistical Orthogonalization: Mapping Equilibrium and Outlier planes
The key principle to keep in mind is that, aestheticization should not conflict with information-loss.Regular data-smoothing techniques [18], [19], [20], although makes the data more presentable, may often leads to information loss [21] intrinsically present as various facets of the data.Often aestheticization is misinterpreted as smoothing to suppress certain features of an image from the visual field.Here, we take an opposite approach.In order to separate the scales of pixel intensities, we divide the rawdata in the image into equilibrium (low-frequency) and outlier (high-frequency) planes that we call statistical orthogonalization1 .From a vector algebra perspective, the dot product of these two planes is identically 0. For the purpose of simplicity, we would synonymously use the terms low-frequency as equilibrium and high-frequency and outlier.
Let's try to understand the problem with a simple story.Suppose two ants of different sizes are sitting on the top of two different elephants.Now, if we want to compare the sizes of the two ants visually, it is not fair to do so when each are on the top of elephants but rather, to first isolate the ants from the elephants before attempting to compare for differences in relative physical dimensions.Since, the physical dimensions of an elephant is several orders of magnitude bigger than that of an ant, it is critically important that, if we want to compare the ants visually it must be done in complete isolation from the context of elephants else, we are at the risk of mismatching the length scales that would bias the judgement behind visual comparison.In the context of USG images, it is typically investigated by a trained radiologist through visual inspection through identification of pre-defined makers.However, we argue that, the presence of multiple intensity scales within the same image lead to a out of scale comparison bias due to significant differences in relative intensity scales.So, irrespective of observational accuracies or training levels of the radiologists, there would always be features that would not be theoretically possible to observe unless we redefine the problem through scale separation as described above.
In order to differentiate the scales, we introduce the concept of statistical-orthogonalization of data that consists of dividing (or decomposing) the original raw-data plane into orthogonal planes on the basis of statistical properties of the data that belong to each plane .We define outlier planes as those, containing the high-intensity pixel values with respect to its neighborhood and kept at corresponding spatial coordinates of the original raw-image in a oneto-one fashion.In the original image, the outlier points are first removed and followed by a neighborhood interpolation in an iterative fashion2 until the rate of outlier discovery becomes negligible (or a point of asymptotic convergence).The threshold and the neighborhood sizes are arbitrarily chosen and each image set is parametrically described in terms of those parameters.

B. Density Color Mapping
Followed by the steps in Sec.II-A, we use a standard density based colorization schema, called ListDensity-Plot [22] implemented in Mathematica [22].Each statistically separated plane (e.g.equilibrium or outlier) is then fed into ListDensityPlot and plotted with Mathematica's standard colorization shema (e.g.Hue, Rainbow, Temperature map etc.).This way the high-intensity or low-intensity pixel values within each plane, are colorized as per the numerical values of the pixels.Details can be found in the documentation of Mathematica software.It is to be noted that, the colorization of pixels are generated within each plane (e.g.equilibrium or outlier) so that all color gradients are attributed as per the properties of that plane.This way, there is a clear separation of scales throgh the pixel value orthogonalization described in Sec.II-A.This scale separation is a highly non-trivial step that can not be achieved simply by direct color density mapping of the pixel values of the raw-images.Also, since each pair of outlier and equilibrium is a function of the threshold (the λ parameter in ref. [17]) and length scale parameters (the # parameter in ref. [17]), it is possible in principle, to create many versions of these equilibrium-outlier statistically-orthogonal imagepairs by varying the parameter values.Here we show only a preliminary example to keep things simple.It is important to note that, the density color maps are defined by the parametric constraints of step in Sec.II-A.Also, the basic reasoning for the selection of density plots is to stabilize the single pixel level data fluctuations that adds an additional layer of stability to the process.

III. Results
The primary motivation of this work has been to bring better clarity, consistency and information-richness to intrinsically noisy USG images.The results are presented under 4 standard Mathematica defined color schemas temperature-map, hue, rainbow and grayscale as demonstrated in Figs. 1, 2, 3, 4 respectively.Each figure should be read in the following order.1) raw-image on the top, 2) equilibrium-image on the bottom-left and, 3) outlier-image on the bottom-right.
Each of those figures represents a certain color spectrum schema as defined ListDensityPlot option in Mathematica [23] computing software.Density plots [24] are a very effective way to visualize scatter point cloud like data sets that we've generated in each of equilibrium and outlier planes as described in Sec.II-A.The outlier color schema is kept the same for the sake of simplicity.If noticed carefully, the eye, teeth of the baby are visible 3 in the image (c) of each figure.The same is not apparent either in the rawimage colorization (e.g.image a) or the equilibrium image colorization (e.g.image b).In image (b) of each figure, the jaw area reveals certain distinct features that were not noticeable in the original image (a) 4 of each figures.This is an evidence of new feature discovery through enhanced clarity.Further granularity can be obtained upon repeat application of the method in Sec.II on the outlier-plane 5 in order to be able to observe the images at much finer details.Careful observation of both the equilibrium and outlier image shows novel features (e.g.eye, ear, jaw, teeth etc.) that were not noticeable otherwise.This is exactly opposite to data-smoothing techniques where features are suppressed and colors are added only to enhance the visual appeals of the image often at the cost of structural and feature information.

IV. Discussion, Future work & Conclusion
We conclude that, even systems with ultra-noisy signatures of data, inherently present in modality like ultrasound-imaging can be re-imagined to be transformed in such a way that, clarity and aesthetic values can be simultaneously enhanced without compromising feature information.We presented few standard Mathematica based schemas to compare and contrast various generic possibilities.However, a plethora of schemas can be applied to both the equilibrium and outlier planes quite independent of each other.It is not a mere coincidence that, we rarely observe an fetal ultrasound image of a baby on the wall our our homes as opposed to that of a newborn.We argue, that the inherent lack of aesthetics in the raw ultrasound-images, is the key reason for this observation as opposed to mere coincidence.The purpose of this work is not only to demonstrate hidden aesthetics in apparently noisy images but rather, to set forth a trend and awareness for the hardware manufacturers to enable optionally available aestheticized images directly from the USG scanner so that the end users have a choice to be able to access features that would otherwise not be possible through raw images.We argue that, this type of advanced near lossless colorization schema without would promote better emotional connection like parent-child bonding even before the birth of the child.There is ample evidence that better mother-child emotional bonding lead to healthier grown of the fetus [25], [26], [27].It is quite apparent from the worked out examples, that overall clarity and information about the image is noticeably enhanced.Amongst the 4 images described here, Fig. 3 (rainbow-schema) and Fig. 4 (grayscale-schema) demonstrate superior image clarity as per the opinion of the authors.We can safely conclude from Figs. 1, 2, 3, 4 that, various colorization schemas do not necessarily lead to equivalent level of clarity enhancements.Advanced schemas would be introduced in future to cover an entire suite of colorization methods and particularly to find out the optimal ones under targeted outcome measures (e.g.soothing to eyes, sharp contrasts etc.).It is important to note that, our method is not necessarily tied to ultrasoundimaging but rather, any kind of image that suffers from clearly distinguishable scale-separation.Adaption of our aestheticization protocol amongst the mainstream ultrasound imaging community and hardware manufacturers would be the key measure of success of this proposed method.In future, this method can be easily applied to other USG modalities e.g.abdominal, carotid artery etc.The principles described here are very generic and not tied to any specific kind of images although the biggest impact area would be noisy images with wide separation of scales.

Figure 1 .
Figure 1.Comparison between original fetal ultrasound image (a) and orthogonalized (equillibrium & outlier) image-pair.Mathematica based temperature-map color-schema is applied to both original (a) and equilibrium (b) planes and & hue schema is applied to outlier (c) plane

Figure 2 .
Figure 2. Comparison between original fetal ultrasound image (a) and orthogonalized (equillibrium & outlier) image-pair.Mathematica based hue color-schema is applied to all the three images in order, original (a), equilibrium (b), & outlier (c)

Figure 3 .
Figure 3.Comparison between original fetal ultrasound image (a) and orthogonalized (equillibrium & outlier) image-pair.Mathematica based rainbow color-schema is applied to both original (a) and equilibrium (b) planes and & hue schema is applied to outlier (c) plane

Figure 4 .
Figure 4. Comparison between original fetal ultrasound image (a) and orthogonalized (equillibrium & outlier) image-pair.Mathematica based grayscale color-schema is applied to all the three images in order, original (a), equilibrium (b), & outlier (c)