Modulation of Point Spread Function for Super-Resolution Imaging

High image resolution is desired in wave-related areas such as ultrasound, acoustics, optics, and electromagnetics. However, the spatial resolution of an imaging system is limited by the spatial frequency of the point spread function (PSF) of the system due to diffraction. In this article, the PSF is modulated in amplitude, phase, or both to increase the spatial frequency to reconstruct super-resolution images of objects or wave sources/fields, where the modulator can be a focused shear wave produced remotely by, for example, a radiation force from a focused Bessel beam or X-wave, or can be a small particle manipulated remotely by a radiation-force (such as acoustic and optical tweezers) or electrical and magnetic forces. A theory of the PSF-modulation method was developed, and computer simulations and experiments were conducted. The result of an ultrasound experiment shows that a pulse-echo (two-way) image reconstructed has a super-resolution (0.65 mm) as compared to the diffraction limit (2.65 mm) using a 0.5-mm-diameter modulator at 1.483-mm wavelength, and the signal-to-noise ratio (SNR) of the image was about 31 dB. If the minimal SNR of a “visible” image is 3, the resolution can be further increased to about 0.19 mm by decreasing the size of the modulator. Another ultrasound experiment shows that a wave source was imaged (one-way) at about 30-dB SNR using the same modulator size and wavelength above. The image clearly separated two 0.5-mm spaced lines, which gives a 7.26-fold higher resolution than that of the diffraction limit (3.63 mm). Although, in theory, the method has no limit on the highest achievable image resolution, in practice, the resolution is limited by noises. Also, a PSF-weighted super-resolution imaging method based on the PSF-modulation method was developed. This method is easier to implement but may have some limitations. Finally, the methods above can be applied to imaging systems of an arbitrary PSF and can produce 4-D super-resolution images. With a proper choice of a modulator (e.g., a quantum dot) and imaging system, nanoscale (a few nanometers) imaging is possible.


I. INTRODUCTION
High spatial image resolution is desired in many waverelated areas such as ultrasound, acoustics, optics, and electromagnetics.However, the spatial resolution of a linear shift invariant (LSI) imaging system is limited by the highest Jian-yu Lu is with the Department of Bioengineering, The University of Toledo, Toledo, Ohio 43606, USA.(e-mail: jian-yu.lu@ieee.org).
In optics, many methods have been developed to improve image resolutions.To improve image axial resolution (along the optical axis), confocal imaging (both illumination and detection are focused on the same spot) [13], 4pi imaging (an object is illuminated by focused lights from opposite directions and also detected from these directions) [14], spatially modulated illumination (SMI) microscopy (creating a standing wave between two objectives to measure the positions of isolated point objects at a resolution better than a half wavelength) [15], and spectral precision distance microscopy (SPDM) (a variant of SMI by adding spectral information) [16] can be used.To obtain a super-resolution in the lateral direction (perpendicular to the optical axis), a number of techniques have been developed.Near-field scanning optical microscopy (NSOM) is a technique where the detector is placed close to the object to catch the evanescent waves [17].Metamaterials such as hyperlens is used to capture the evanescent waves by placing the object to be imaged close to the materials [18].Structured illumination microscopy (SIM) uses different patterns of the light to illuminate an object either in far field (increase resolution by 2 folds) [19] or near field (can increase the resolution much more) [20].The SIM can also be applied to thermal imaging [21].Stimulated emission depletion (STED) is a fluorescent imaging technique that uses a probing beam to excite the molecules to fluoresce and then a STED beam is applied to deplete the out ring of the fluorescence, leaving a very small center part of the fluorescence to form a superresolution image [22].A similar method as the STED is called ground state depletion (GSD), which lowers light intensity and thus has less photo bleaching effects to the objects to be imaged [23].Another method that is an inverse of the STED using photobleaching to suppress the center area of the heating of the optical beam and then using subtraction to get an equivalent localized heating that is detected by a highfrequency ultrasound (40 MHz) (photoacoustics) without resorting to fluorescence to get super-resolution in both lateral and axial directions [24] (the resolution improvement depends on the photobleaching characteristics of the object to be imaged).Single molecule localization microscopy (SMLM) [25]- [26] represents a group of techniques using different ways to induce a stochastic emission of the fluorophores so that images can be formed by accumulating different sets of localized molecules obtained at different time.This includes photoactivated localization microscopy (PALM) (using proteins that are photoactivatable, photoconvertible or photoswitchable) [27], points accumulation for imaging in nanoscale topography (PAINT) (using reversibly-binding fluorescent agents) [28], and stochastic optical reconstruction microscopy (STORM) (using fluorescent organic dyes with a specific buffer) [29].Another way to localize molecules is to use the super-resolution optical fluctuation imaging (SOFI) [30], where the image is obtained by analyzing the statistics of the temporal fluctuations of a series of images.The principle of SMLM can also be applied to imaging that does not need fluorescence [31].
In this paper, a general PSF-modulation method was developed for super-resolution imaging [50].In the method, the PSF of an imaging system is modulated so that the resulting PSF has a higher spatial frequency.The modulated PSF is then scanned through an object (see Fig. 1(a)) or is processed with a camera (see Fig. 1(b)) or a holographic imaging system [51]- [52] to get a super-resolution image.In addition, a variant of this method called PSF-weighted super-resolution imaging was developed (see Appendix B), where, like in the camera or the holographic imaging system, the modulator itself is scanned or moved through an object to form an image.
In Appendix A at the end of this paper, ideas for practical implementations of the PSF-modulation method are presented, and the theory of the PSF-modulation method is extended to the camera imaging system (see Fig. 1(b)) and to the optical imaging systems where the intensity of the light is measured.Appendix A includes the following: (i) Using a quantum dot [53] [54] as a modulator, it is possible to get super-resolution images with a resolution (a few nanometers) which is similar to that of a scanning electron microscope (SEM) [55] but without needing a vacuum if the quantum dot could emit a constant light intensity during imaging (see Appendix A.1). (ii) Using a conventional medical ultrasound imaging system with a linear or phased array transducer, it is possible to obtain superresolution images by using an interference pattern of shear waves as a modulator (see Appendix A.2 and a related example that uses a two-dimensional (2D) or annular array in the last two paragraphs of Section V "Methods and Results").(iii) Using microbubbles in the ULM [45]- [47] and nanodroplets [49] as modulators, it is possible to get super-resolution images of blood vessel walls and their surrounding soft tissues at a microscopic resolution deep in the tissues and get nanoscale images of the interior of a cell respectively if the dynamic range and the signal-to-noise ratio (SNR) of the imaging system are large enough (see Appendix A.3). (iv) The PSFmodulation theory for super-resolution imaging is extended to the camera imaging system (see Fig. 1(b)) that can represent bright-field optical microscopes, mobile optical microscopes, and acoustical cameras (see Appendix A.4) and to the optical imaging systems where the intensity of the light is measured (see Appendix A.5). (v) Potential applications of the PSFmodulation method to interferometer system, in-line digital holography [51] [52], and optical coherence tomography (OCT) [56] for super-resolution imaging are also given in Appendix A.5. (vi) Other potential applications of the PSF-modulation method for super-resolution imaging include magnetic resonance imaging (MRI) [57], nondestructive evaluations (NDE) of materials, scanning acoustical microscope (SAM) [58][59], photoacoustic imaging [60], and underwater acoustics [61] (see Appendix A. 6).
Paper organization: The theory of the PSF-modulation method is presented in Section II, followed by two examples in Section III.Computer simulation and experiment procedures are given in Section IV and methods and results are in Section V. Finally, discussion and conclusion are in Section VI and VII respectively, followed by the Appendix.

II. THEORETICAL PRELIMIARIES
Let's consider a wave field ( ; ) Φ T r t (such as ultrasound, acoustics, electromagnetic, and light wave) that is produced by a wave generator (such as an ultrasound transducer, loudspeaker, electromagnetic antenna, and optical light source) (see Fig. 1(a)), where ( , , ) = r x y z is a point in space, t is the time, the superscript " T " means "transmit", and ( ; ) Φ T r t is a convolution of a drive signal ( )  s t and the impulse response ( ; ) h r t of the wave generator in terms of time in a linear time invariant (LTI) system.If the wave generator also is used as a receiver (notice that the receiver can be a separate device), the receiver response can be written as ( ; ) Φ R r t , where the superscript " R " represents "receive".
At each given time t , in an LSI imaging system (notice that many practical imaging systems in various areas of science and engineering such as ultrasound and optics can be described or approximately described by an LSI system), the received signal from an object ( ) f r that represents wave scattering coefficients or other properties of the object is given by the following convolution [1]: where the superscript "PE" means "pulse-echo" or "two-way", ' r is an integration variable over the space, * r represents a convolution in terms of r , and at each fixed spatial position r , we have: where * t represents a convolution in terms of time t .If ( ) f r is a point object, i.e., ( )  ( ) δ = f r r , where ( ) δ r is the Dirac-Delta function [62], from Eq. ( 1), one obtains the PSF of the imaging system at each given time t : ' PSF ( ; ) ( '; ) ( ') ' ( ; ) Using Eq. ( 3), Eq. ( 1) can be written as: Taking a spatial Fourier transform on both sides of Eq. ( 4), one obtains: ( ; ) PSF ( ; ) ( )  5), it is clear that the maximum spatial frequency of the image ( ; ) PE R r t is limited by that of PSF ( ; ) PE r t due to wave diffraction, which is a fundamental limit to the spatial resolution of an imaging system.
To increase the bandwidth of the imaging system, the PSF function can be multiplied by a complex modulation function ( ) m r (the physical meaning of ( ) m r is to introduce a disturbance to an existing PSF of the imaging system by amplitude, phase, or both).The resulting image is given by: where the subscript "m" of "PE" means "modulation", and ( ) PE C t is independent of the spatial variable r and is a constant at any given time t because the relative position between the modulator and the PSF is fixed (see Fig. 1(a)): γ PE m r represents the scattering or reflection coefficient of the modulator ( )  m r in the pulse-echo system.Taking a spatial Fourier transform on both sides of Eq. ( 6), we have: r t r t m r , of the imaging system is increased, making it feasible to reconstruct super-resolution images with methods that are suitable for specific applications in different areas of science and engineering.

If ( ; )
f r t represents the spatial field (a distribution of the wave in space r in a media, for example, the acoustical waves produced by light or electromagnetic heating in photoacoustic imaging [60], waves transmitted through an object, and the waves used to illuminate and then scattered or reflected from the object) of a wave source at any given time t , a superresolution image of the spatial field also can be reconstructed.In this case, only a receiver is needed and the imaging is a "one-way" process.Similar to Eqs. ( 4) and ( 6), images without and with a modulator can be obtained respectively as follows: where , * r t represents a convolution with respect to both r and t , and where ( ) γ R m r represents the scattering or reflection coefficient of the modulator ( ) m r .From Eqs. (10) and (11), it is clear that ( ) γ R m r can be viewed as a part of the modulator.Similar to Eq. (3), the PSF of the receiver can be obtained: r t r r t t r t dr dt r t . ( Notice that Eqs. ( 4), ( 6), (9), and (10) work for the camera imaging system in Fig. 1(b) too if the LSI condition is met (see the theory in Appendix A.4. "Camera Imaging System").Imaging systems with modulation of the point spread function (PSF).(a) Modulation of the PSF of an imaging system that uses a spherically focused wave (such as ultrasound, optical, and electromagnetic waves) for four-dimensional (4D) super-resolution imaging.The modulator is on the wave propagation axis, z , and is centered at the focal distance F .(b) A camera imaging system with a modulator on the object plane.The image A'B' is formed on the image plane with an object AB (arrow) on the object plane.The camera and object are fixed in space while the modulator is scanned over the object plane to obtain a super-resolution image (see details in Appendix A.4. "Camera Imaging System").
Also notice that the position of the modulator ( ) m r is assumed to be fixed relative to the PSF in both Eqs. ( 6) and (10).However, this condition may not be met in some applications.For example, when a modulator such as a charged or magnetic nanoparticle [63] is moved by an electrical or magnetic force to pass through a cell via cellular pores or around cells in an extracellular matrix along different paths for super-resolution imaging, the position of the modulator relative to the PSF may change.Similarly, when a charged particle is used as a modulator to get a super-resolution image of the topological structure of the gel or materials embedded in the gel in electrophoresis [64], the position of the modulator relative to the PSF may also change as the modulator moves through the gel at a certain speed.Although the PSF can change rapidly with time at the temporal frequency of the imaging wave, the magnitude of the PSF does not change quickly over both time (due to a limited temporal bandwidth) and space (due to a poor diffraction-limited resolution).Thus, if the shifts of the modulator are within the range where the magnitude of the PSF does not change significantly, both Eqs. ( 6) and (10) can still be used since in most cases, the magnitude-related quantities such as averaged absolute values and analytic envelope, instead of the radio frequency (RF) carrier wave that fluctuates quickly at the temporal frequency of the imaging wave, are of interest in imaging.To ensure that ( ) PE C t in Eq. ( 6) is independent of r and thus can be removed, it is necessary to shift the image pixels along the time direction to compensate for the shifts of the modulator in the z (axial) direction (see Fig. 1) temporally (notice that the position of the modulator can be tracked in both time and space as it is normally done for microbubbles in the ULM [8]- [10]).After removing ( ) PE C t , the positions of the pixels should be shifted back to where the modulator was.Or, if the "I" (in phase) and "Q" (quadrature) components of the RF signal are available, they can be used directly to remove ( ) PE C t without the need of the shifts above.The procedure above can produce a superresolution image on a curved surface that represents the trajectories of the nanoparticle passing through the cell, the extracelluar matrix around the cells, the gel, or the materials embedded in the gel.
III. EXAMPLES Eqs. ( 6) and (10) are the basis for four-dimensional (4D) ( , , ; ) x y z t super-resolution imaging with PSF modulation.In general, the modulation function ( )  m r is complex and can produce both amplitude and phase modulations to the imaging wave, and one is free to choose as long as corresponding image reconstruction methods can be developed.However, some choices of ( ) m r can simplify the implementation of the PSFmodulation method and image reconstructions.Below we give two ( ) m r examples that are relatively simple.

A. Amplitude Modulation
The first example is to assume that ( ) m r is a real function that produces an amplitude modulation to the imaging wave: where 1 ( , , F , F is the focal length, and a is the radius of the modulator (see Fig. 1(a)).Inserting Eq. ( 13) into Eq.( 6), and then subtracting the result from Eq. ( 4), a 2D super-resolution C-mode image (in the − x y plane at = z F ) can be reconstructed by where the subscript "sub" on "PE" means "subtracted", and ( ) C t is due to the reflections from the modulator 1 ( ) m r (with r mr ) and is given by (see Eq. ( 7)): Similarly, 2D super-resolution C-mode images of the field of a wave source (one-way imaging) using the modulator in Eq. ( 13) can be reconstructed by inserting Eq. ( 13) into Eq.(10) and then subtracting the result from Eq. ( 9): where ( ; ) R C r t is given by Eq. (11) and is part of the reconstructed image of the wave source/field.

B. Phase Modulation
The second example is a modulator for phase modulation: where 1 = − i and 0 0 2 φ π < < is a constant.As in the amplitude modulation, 2D super-resolution C-mode pulse-echo images can be reconstructed by Eq. ( 14) with (see Eq. ( 15)): For 2D C-mode one-way super-resolution imaging of the field of a wave source, Eq. ( 17) can be used (see Eq. ( 18)):

IV. COMPUTER SIMULATION AND EXPERIMENT PROCESURES
The computer simulations were based on the limiteddiffraction array beam method developed in Ref. [65] to obtain both ( ; ) Φ T r t and ( ; ) Φ R r t , and were implemented using the C programming language [66] on a Linux operating system.Using the modulators given in Eqs. ( 13) and ( 19), 2D superresolution C-mode images can be reconstructed with Eqs.(14) or (17), depending on pulse-echo or wave source/field imaging.In the simulations, the following parameters were assumed: the modulator radius a = 0.The experiments were performed according to the block diagram in Fig. 2. In the experiments, a 1-cycle sine wave (pulse) of 1 MHz and about 0.5 Vpp ("pp" means "peak-topeak") amplitude was produced by a function generator (HP8116A, Hewlett-Packard Company, CA, USA) and then amplified by an RF power amplifier (ENI2100L, Electronics and Innovation, Ltd., NY, USA) or two home-made RF power amplifiers of opposite polarities (for linear array experiment only).The amplified signal was used to drive a transducer (V302, Panametrics, MA, USA) that had 1-MHz center frequency, about 65% one-way relative bandwidth, and a 25.4mm diameter for pulse-echo imaging, or drive another transducer for wave source/field imaging.To focus the waves, an acoustical lens that was made of acrylic and had a focal length F of about 50 mm (the measured F was about 44.12 mm) was attached to the front surface of the 1-MHz Panametrics transducer.Echoes from an object or the sound emitted from a wave source were received by the Panametrics transducer.The received signals were amplified by a 40-dB preamplifier and then filtered by a home-made 6-pole, constant-k, and 1-MHz center frequency band-pass filter of about 33% relative bandwidth before being digitized by a 12bit digitizer at 20 MegaSamples/s for 512 samples after a fixed time delay.After each signal was received, the motors moved the object or the wave source to be imaged to the next position in the − x y plane at a fixed axial distance z = F with a step size of 0.1 mm along both x and y axes.At each ( , ) x y position, an unsynchronized trigger signal was generated from the motors and was synchronized to the clock of the digitizer before being sent to the function generator to produce the 1cycle pulse.The process was repeated until all data were collected for each image with and without a modulator (a photo and a description of the modulator used in the experiments will be given in Fig. 3(a) in Section V "Methods and Results").
2D C-mode super-resolution images were reconstructed with Eqs. ( 14) (pulse-echo imaging) or ( 17) (wave source/field imaging) from the experiment data collected with and without the modulator.In the image reconstructions, the DC component ( ) C t of the images at each temporal sample point was removed, and the final images were obtained by summing the absolute values of the images incoherently over 11 sample points (i.e., over a duration of about a half cycle of the 1-MHz signal).In the simulations, the incoherent summation was not necessary since a CW signal was assumed, the digital objects were infinitely thin, and the modulator and the objects were at the same distance from the transducer.However, in the experiments, the incoherent summation was needed to average out the variations of the images obtained at different temporal sample points since a short pulse was used, the objects were a few millimeters thick, different parts of the objects had a slightly different distance from the transducer, the modulator on the supporting wires (Fig. 3(a)) vibrated randomly in the z direction when the water was disturbed by the scanning of the objects or wave sources, and the modulator was about 0.5 mm away from the objects.Despite of the deviations from ideal (theoretical) conditions, good super-resolution images with a high SNR were reconstructed in the experiments (see Section V below).

Fig. 2.
Block diagram of the experiments.(a) A photo of the water tank, a transducer, a modulator, and an object.The transducer and the modulator were mounted on one axis, and the object was on another that can be moved in the xy plane by motors.(b) Block diagram of the imaging system.One cycle (pulse) of a 1-MHz sine-wave signal from the power amplifier was used to drive either the transducer (pulse-echo imaging) or a wave source (one-way imaging).The received signals were filtered by a band-pass filter of 1-MHz center frequency and a -6dB fractional bandwidth of 33% before being digitized at 20 MegaSamples/s and a 12-bit resolution.
V. METHODS AND RESULTS Fig. 3(a) is a photo of the modulator that can be approximated with Eq. ( 13) for amplitude modulation.The modulator was made of a 0.5-mm diameter and 1-mm long copper wire.It was glued to the center of a cross of a pair of 85μm diameter copper supporting wires that were mounted on a copper ring of 38.1-mm inner diameter and were tensioned via two pairs of small springs.The copper ring had three small clips for mounting and removing the modulator without affecting the transducer alignment.Figs.3(b) and 3(c) are simulated pulse-echo (two-way) C-mode images of a point object (a geometrical point without a physical dimension) with (see Eq. ( 6)) and without (see Eq. ( 4)) the modulator (see Eq. ( 13) with a = 0.25 mm) respectively.Fig. 3(d) is the superresolution image reconstructed with Eq. ( 14).° [68] in the experiments), F is the focal length of the transducer (50 mm in the simulations and about 44.12 mm in the experiments), and D = 25.4 mm is the diameter of the transducer.Using these parameters, the diffraction-limited resolutions calculated with Eq. ( 22) are about 3.0384 mm and 2.65 mm for the simulations and the experiments respectively.Notice that the 44.12-mm focal length of the experiments was estimated from Eq. ( 22) using the measured diffraction-limited resolution (2.65 mm).
Comparing the images in Figs.3(d) and 3(h) with the diffraction-limited images in Figs.3(c) and 3(g), as well as from Fig. 4, it is clear that super-resolution images were reconstructed with the PSF-modulation method at an SNR of about 31 dB.Fig. 5 shows pulse-echo imaging of an "L"-shaped object that was consisted of 11 point objects.Fig. 5(a) shows the dimensions of the object.On the left in the vertical direction, there were 6 point objects with spacing of 2.5, 2.0, 1.5, 1.0, and 0.5 mm respectively from top to bottom.In the horizontal direction, there were 5 point objects with spacing of 0.5, 1.0, 1.5, and 2.0 mm from left to right respectively.The vertical distance between the right-most two point objects was 0.5 mm.respectively with and without the modulator (the modulator used was the same as that in Figs. 3 and 5, and was about 0.5 mm away from the surface of the array).Fig. 6(f) is the superresolution image reconstructed using Eq. ( 17).
The one-way diffraction-limited FWHM resolution derived from Eq. ( 36) of Ref. [67] is given by:   In both Figs.7(d) and 7(e), none of the lines is visible.However, in Fig. 7(f), even the lines separated by 0.5 mm are visible, which means that the image resolution was about 0.5 mm that was 7.26 folds better than the 3.63-mm diffraction limited resolution calculated by Eq. ( 23) and also was better than a half of the 1.4826-mm wavelength.In addition to the line patterns, Fig. 7(f) shows that the transducer had a complicated vibration mode.The SNR estimated from a line plotted across a line pattern of Fig. 7(f) is greater than 30 dB.The lines separated by 0.5 mm can be clearly seen in (f), suggesting that the resolution was about 0.5 mm.Fig. 8 shows the simulated results of a point object (see Fig. 3) and an "L"-shaped object (see Fig. 5) using the modulator given in Eq. ( 19) to modulate the phase of the PSF with 8(a)-8(c) respectively, except that an "L"-shaped object was imaged.As in Fig. 5, all of the 11 points of the "L"-shaped object were geometrical points without a physical dimension.From Figs. 8(c) and 8(f), it is clear that super-resolution images can be reconstructed with a phase modulation.The line plot of Fig. 8(c) is the same as that of Fig. 3(d) and is given in Fig. 4 (red dash-dotted line), showing a resolution of 0.5 mm.Although in this paper there are no experiment results for super-resolution imaging with a phase modulation as in Fig. 8, it is possible to obtain them in the future with an implementation shown in Fig. 9 that is briefly described as follows.An annular array [69] or a 2D array transducer can be driven by a long sine wave (RF signal) at the center frequency of the transducer to produce a narrowband focused Bessel beam [69]- [70] or X wave [71].The RF sine wave can be modulated in amplitude by a low-frequency sine wave or square wave of a few cycles (turn the transmitter on and off periodically for a few times to produce the square-wave modulation).Around the focal distance, as was demonstrated in Figs. 1 and 3 of Ref. [72] and Eqs. ( 31)-(33) of Ref. [67], the focused Bessel beam or X wave can remotely produce a cylindrical ring [73]- [74] of radiation force that in turn can produce a cylindrical ring of narrowband (centered at the modulation frequency) shear wave, which will propagate both inward and outward from the ring.The inward propagating shear wave will focus around the center of the ring on the z axis to produce a sharp peak [75] and modulate the phase of the short (broadband) imaging wave pulse (a longitudinal wave such as a focused plane wave or Gaussian beam) for 2D or 3D (three-dimensional) super-resolution imaging with Eqs. ( 14), (19), and (20) if the highest spatial frequency of the PSF is increased due to the shear wave modulation (the width of the peak of the shear wave on the center of the cylindrical ring will be close to a half of the shear-wave wavelength that can be small to produce a high-resolution image if the modulation frequency is high and the shear-wave speed is low).Notice that the same annular [69] or 2D array transducer can be used to produce both the imaging wave and the cylindrical ring of radiation force, and the Bessel beam or X wave can be truncated on the transducer aperture to increase the total transmit power [72] to produce a stronger radiation force to increase SNR.Also, a one-dimensional (1D) linear/phased array transducer can be used to produce shear waves for phase modulation to reconstruct super-resolution images (see an example given in Appendix A.2 "Phase Modulation with a 1D Array").
Notice that unlike physical modulators such as the one used in the experiments (Figs.2-7), when shear wave is used as a modulator, there will be sidelobes in imaging (also see Appendix A.2).In Fig. 9, although a cylindrical ring of shear wave produced by a focused [67][72] Bessel beam [69]- [70] or X wave [71] can focus on the z axis for phase modulation, high-frequency shear wave will have a high attenuation in materials such as biological soft tissues and thus the magnitude of the mainlobe of the shear wave may be reduced and the sidelobes are relatively increased.To reduce the sidelobes, rotated higher-order Bessel beams (see Figs. 6-7 of Ref. [72]) or X waves that produce unsymmetrical radiation forces could be used [76].Or, the images obtained with the shear waves produced by the radiation force of a focused plane wave and a focused zeroth-order Bessel beam or X wave can be subtracted to get an image of lower sidelobes in a way similar to STED [22] while minimizing the attenuation by reducing shear wave propagation distance (also, the focused plane wave can be replaced with a Bessel beam of a small scaling parameter if needed).In addition, two shear waves of low and high frequencies respectively can be used to reduce the sidelobes through a coherent subtraction since the high-frequency shear wave has similar sidelobes as those of the low-frequency but does not have a mainlobe or has a small mainlobe magnitude due to a high attenuation (also see Appendix A.2).

Fig. 9.
This figure is the same as Fig. 1(a) except that the modulator will produce a phase modulation.A cylindrical ring of shear wave can be produced by a radiation force that is produced by a focused Bessel beam or X wave (see Figs. 1 and 3 of Ref. [72] and Eqs. ( 31)- (33) of Ref [67]).The shear wave can focus on the z axis to modulate the phase of the PSF of the imaging system to produce a 2D or 3D superresolution image.

A.
Image Resolution It is clear from Eqs. (6) (pulse-echo imaging) and ( 10) (wave source/field or one-way imaging) that the maximum spatial frequency of the modulated signal is increased due to a convolution of the PSF of the original imaging system with the modulation function in the spatial frequency domain.Also, if the maximum spatial frequency of the modulator is much larger than that of the original PSF, the maximum spatial frequency of the modulated PSF will be dominated by the modulator.Thus, if the radius a of the modulators in Eqs. ( 13) and (19) decreases, the resolution of the super-resolution images reconstructed will increase (see Figs. 3, 4 and 8 where the resolution of the super-resolution images reconstructed is close to 0.5 mm, the diameter of the modulator).If 0 → a , at any given time t , the subtracted PSF in Eqs. ( 15), ( 18), (20), and (21) will be proportional to a 3D Delta function ( )   δ r [62] that has a very large spatial bandwidth.Since the convolution of a Delta function with an object is the object itself (see Eqs. ( 14) and ( 17)), in theory, the maximum resolution that can be achieved with the PSF-modulation method is unlimited.
In Eqs. ( 6) and ( 10), the modulator ( ) m r for either the amplitude or phase modulation of the PSF should be within ( ) f r or ( ; ) f r t , which is the case for the simulations in Figs. 3, 4, 5, and 8, and the example in Fig. 9.However, in the experiments shown in Figs.3-7, the modulator was about 0.5 mm away from the objects to be imaged.This will reduce the image resolution since the waves will diffract after passing the modulator, and as the distance between the modulator and the object increases, the image resolution will decrease.

B.
Noise Although in theory the resolution of images reconstructed with the PSF-modulation method is unlimited, in practice, imaging systems always have noises.From Eqs. ( 15), ( 18), (20), and (21), it is clear that as the radius a of the modulator decreases, the modulation area decreases in proportional to 2  a and thus the signals received (see Eqs. ( 14) and ( 17)) will decrease proportionally.If the SNR is too low to distinguish features in the reconstructed images, further reduction of a will not lead to a better image quality.This sets a limit to the maximum image resolution.Fortunately, the experiment results in Figs.3-7 have a high SNR, thus there is room to further increase the image resolution by decreasing a from 0.5 mm that was already about 1/3 of the wavelength used.
To increase the SNR of the reconstructed super-resolution images, the magnitude of the PSF should be maximized in the area of the modulator (see Eqs. ( 15), ( 18), (20), and ( 21)), although in principle, the PSF-modulation method can work with an arbitrary PSF (see Eqs. ( 14) and ( 17)).To increase the PSF magnitude, both of the transmit and receive waves can be focused for pulse-echo imaging, and the receiver can be focused for wave source/field imaging, as were the cases in the experiments in Figs.3-7.Also, the modulation functions will affect the received signals.In the amplitude modulation given in Eq. ( 13), if the modulator is transparent, i.e., is a constant, the SNR will decrease as 0 1  → m (more transparent) (see Eqs. ( 15) and ( 18)).For the phase modulation in Eq. ( 19), as 0 0 φ → , the SNR will decrease because  20) and ( 21)).Another way to increase the SNR is to use a more complicated modulator pattern ( ) m r (see the last paragraph in Appendix A1.)

C.
Dynamic Range of the Receiver In some implementations of the PSF-modulation method, there may be a subtraction involved, as is in Eqs. ( 14) and (17).This means that the subtracted signals may be smaller than those before the subtraction, especially at a higher image resolution, i.e., a smaller a in Eqs. ( 13) and (19).For other implementations such as the high-pass filtering method in the second paragraph of Section VI.F "Motion Artifacts", although no subtraction is used, the filtered signals may still be small as compare to those before the filtering.In these cases, the receiver and its associated digitizer should have a large enough dynamic range so that the small difference or filtered signals will not be buried in the noise or lost in the quantization error.Thus, in addition to the noise, dynamic range of a receiver also may limit the maximum resolution of the reconstructed images.For the experiments in Figs.2-7, the digitizer has 12-bit resolution, which gives a dynamic range of about 72 dB.

D.
System PSF and 2D Imaging with 1D Arrays From Eqs. ( 2) and (3), it is clear that in principle the PSFmodulation method works on an arbitrary transmit and receive beam or wave configuration in different wave-related areas such as ultrasound, electromagnetic, and optics, although some transmit and receive configurations work better than others in terms of the SNR and ease of implementations.This means that the PSF-modulation method can be used with many of the current transmit/receive confirmations of the ultrasound and optical imaging systems and other configurations developed in the future.In this paper, the PSF-modulation method has been demonstrated for both pulse-echo (Figs. 3, 5, and 8) and wave source/field (Figs. 6 and 7) imaging.In general, the PSF may rotate and/or change its shape and value while moving from one place to another.If the PSF changes, the images reconstructed may be affected.
Since the PSF can be arbitrary, in addition to the imaging systems where a circularly-symmetric focused transducer was used (Figs. 3, 5, 6, 7, and 8), the PSF-modulation method can be applied to commercial medical ultrasound imaging systems that use 1D linear, phased, and curved array transducers to translate or steer the beams electronically to form 2D B-mode images (B-mode image is formed in an − x z plane, see the coordinates in Fig. 1(a)).However, the super-resolution images reconstructed with a 1D array may have a lower SNR since the focusing gain of the 1D array may be less than that of a circularly-symmetric focused transducer or a 2D array.An example of using a 1D linear or phased array for superresolution imaging is given in Appendix A.2.

E.
Curved Image Surface, 3D and 4D Imaging At each given time t , from the 3D convolution over the spatial variable ( , , ) = r x y z in Eqs. ( 6) and (10), it is clear that the PSF-modulation method is not restricted to the 2D C-mode imaging given in the examples in Figs. 3, 5-8.The image "plane" can be an arbitrarily curved surface, or the image can be a 3D volume.Changing the time, t , a 4D super-resolution image can be reconstructed, although a 4D imaging may take a long time since the modulated PSF must move over the entire 3D volume of the object to be imaged to perform the 3D spatial convolution.
If we restrict the spatial convolution to 2D in the − x y plane in ( ; ) m PE R r t , 3D images can be reconstructed with a pulse-echo imaging system using the time as the third dimension.In this case, the axial resolution (the resolution along the wave propagation direction z ) of the 3D image is not affected by the PSF modulation if the modulator is only a function of the space, i.e., ( ) = m m r ; rather, it is affected by the temporal bandwidth of the imaging system (in Figs. 3 and  5, a short 1-cycle sine-wave pulse was used).If the modulator also is a function of time t , i.e., ( ; ) = m m r t , the axial resolution can be increased.However, in practice, there is always a limit on how much the temporal bandwidth of an imaging system can be increased to get higher axial and lateral resolutions due to factors such as frequency-dependent attenuation [33].
To keep a super-resolution in the lateral direction over an axial range in the 3D pulse-echo imaging above, the modulator should cover the axial range.In fact, except for a small number − f that is close to 1, a focused beam has a certain DOF (or depth of focus).When a focused [67][72] Bessel beam [69]- [70] or X wave [71] is used to produce a radiation force [73]- [74] in Fig. 9, it will not be a thin ring, but will be a cylindrical ring with the height of the cylinder equals to the DOF (see Figs. 3, 5, 7, and 9 of Ref. [72]).This cylindrical ring of radiation force will produce a focused shear wave to modulate the phase of the imaging wave to obtain 3D superresolution images within the DOF [75].In this case, data acquisition for 3D C-mode super-resolution imaging will be the same as that for a 2D C-mode imaging since the signals are usually collected along the time direction anyway, as was the case for Figs. 3 and 5, where 512 samples were acquired for each temporal signal.Similarly, when a 1D array is used, 2D super-resolution images can be obtained (see Appendix A.2).
For wave source/field imaging, the 3D convolution in Eq. ( 10) will produce a 3D super-resolution image of the wave field in media such as biological soft tissues at a given time t , modified by the receiver impulse response PSF ( ; ) R r t due to the convolution in terms of time.As the time changes, a 4D image can be obtained to show an evolution of the 3D wave field over the time.Getting such a complete wave field in space at a high resolution will help to solve the inverse scattering problems in imaging [77].If we restrict the convolution to 2D in the − x y plane at a fixed z in ( ; ) m R R r t and use a short pulse to excite the wave generator as was done in Figs. 6 and 7, a 2D C-mode super-resolution image of the wave field at distance z and its 3D time evolution can be obtained.

F.
Motion Artifacts Using Eq. ( 14) or (17) to reconstruct a super-resolution image, the subtraction can be done either pixel-by-pixel or on the entire image.If the object is stationary, both methods will produce the same result.However, for moving objects such as a human heart, the result may be different.Thus, to minimize the motion artifacts, one should reconstruct the image pixel by pixel.Notice that there is no particular order in choosing a pixel, as long as the pixels cover the entire image.Also, multiple pixels can be reconstructed at once to increase the imaging speed using the camera in Fig. 1

(b) (Appendix A.4).
Another way to reduce motion artifacts is to let 0 → a (using a modulator of a small size for a high image resolution) in Eqs. ( 13) and (19).In this case, 1 ( ) − m r will resemble a Delta function ( ) δ r , as mentioned in Section VI.A "Image Resolution" above.Thus, the modulated PSF in Eq. ( 6) and (10) will have a very high spatial frequency as compared to the original PSF and super-resolution images can be reconstructed approximately (missing low-frequency components) by a highpass (HP) filtering of the images to remove the contribution of the original PSF without a subtraction.However, as with the subtraction, since the magnitude of the high-frequency components becomes smaller as 0 → a , the SNR will reduce.Also, since the low-frequency components are dominated by the original PSF and are almost unchanged as 0 → a , the imaging system should have a large dynamic range to handle the small high-frequency signals.
Other method to reduce motion artifacts for superresolution imaging is to use a wave-emitting modulator such as a quantum dot [53][54] (see Appendix A.1) or a small modulator embedded in a wave-absorbing sheet (see the SAM [58] [59] in the second paragraph in Appendix A.6) to implement the 1 ( ) − m r function with Eq. ( 18), (28), or (29) without a subtraction.The advantage of the method is that it does not require a very large dynamic range to reconstruct super-resolution images at a high resolution (can be a few nanometers in the case of a quantum dot [53] [54]).

G. Super-Resolution Imaging of Flow
The spatial resolution of blood flow imaging is limited by the diffraction limit of imaging systems, except for ULM [45].Thus, the PSF-modulation method will also benefit the blood flow imaging if the super-resolution images can be obtained fast enough so that the changes of speckles or the motioninduced phase change of the received signals can be tracked.As mentioned above, the PSF-modulation method can be implemented one or multiple image pixels at a time to speed up the imaging process and reduce the motion artifacts.

H. Contrast Mechanism of Images
In medical ultrasound imaging, the contrast mechanism of the reconstructed super-resolution images can be backscattering coefficients (pulse-echo imaging, Figs 3 and 5) and ultrasound pressure field distribution (wave source/field imaging, Figs. 6   and 7) using an amplitude modulation.In phase modulation using a shear wave (Figs. 8 and 9, and Appendix A.2), the image contrast will be related to the mechanical properties of the objects such as biological soft tissues.In general, the image contrast depends on both modulation methods and objects to be imaged.If the modulator changes its shape and value during imaging, the image contrast will include such changes.

VII. CONCLUSION
Methods (PSF modulation, and PSF-weighted in Appendix B) for 4D super-resolution imaging of a passive object or a wave source/field were developed by modulating the PSF of an LSI and LTI imaging system.Theories of the methods were presented, and computer simulations and experiments were conducted.Although in theory the spatial resolution of the reconstructed images is unlimited, the resolution is limited by the SNR in practice.The modulator can be created remotely or can be a physical object manipulated remotely.The methods can be applied to areas such as acoustics, ultrasound, optics, and electromagentics.With a proper selection of a modulator and an imaging system, nanoscale imaging is possible with the methods.

A.
Modulation Methods and Implementations In this subsection, some ideas of practical implementations of the PSF-modulation method for super-resolution imaging will be presented, and theories for camera imaging systems (see Fig. 1(b)) and optical imaging systems will be developed.

1) General Implementation
In general, the modulation function ( ) m r in Eqs. ( 6) and (10) can be an arbitrary complex function, as long as corresponding image reconstruction methods can be developed.However, as mentioned above, different ( ) m r can affect the complexity of practical implementations of the method and subsequent image reconstructions.Thus, finding a suitable method to modulate the PSF of an imaging system of interest is a key to implement the PSF-modulation method in practice.
In this paper, two examples of the PSF-modulation method were demonstrated: amplitude (Eq.( 13)) and phase (Eq.( 19)) modulations.These examples are relatively simple to implement.In general, both amplitude and phase modulations can be present at the same time.In fact, in the experiments (Figs.3-7), a small piece of copper wire of 0.5 mm in diameter and 1 mm in length was used as a modulator.The copper wire had a higher speed of sound than that of water and thus a phase modulation was introduced in addition to the amplitude modulation.However, the phase modulation was small because only a small amount of ultrasound can penetrate the copper wire due to an impedance mismatch.Also, if '( ) 1 ( ) = − m r m r in Eqs. ( 13), ( 15), ( 18), ( 19), (20), and ( 21) is used as a modulator, i.e., only allow incident wave to pass through (or reflect from) a small area of radius a and block (or absorb) the wave elsewhere, or the modulator is a small particle that by itself emits waves (such as a fluorophore [22][27], a gas-filled vesicle that produces bursting sound [78], and a quantum dot [53][54]), the subtraction in Eqs. ( 14) and ( 17) is not necessary and the imaging system could be simplified.Since particles such as quantum dots can be very small (around 3 nm for blue light and 8 nm for red light), super-resolution images of objects such as thin films can be reconstructed at a resolution that is close to the size of the quantum dot using the imaging system in either Fig. 1

(a) or 1(b).
To understand how the super-resolution imaging works with quantum dots as modulators, let us assume that a uniform thin film with two small defects separated by a few nanometers apart on one of its surfaces is placed between a quantum dot (the quantum dot is on the surface that contains the defects) and a detector that is tuned to receive the specific emitting wavelength of the quantum dot (assuming that the quantum dot emits light with a constant intensity during imaging, otherwise, the variations of the intensity or the blinking of the quantum dot should be compensated).If the quantum dot is moved on the surface of the thin film by an external force remotely (such as a radiation force [73][74], magnetic/electromagnetic force for magnetic quantum dots [54], or other forces), the light received from the quantum dot will be dimmed twice by the two defects if the defects absorb part of the light when the quantum dot is moved over them (since the quantum dot is very close to the defects, a large portion of the light energy will be absorbed) (here we assume that the receiver can track the motion of the quantum dot and keep the relative position between the quantum dot and the receiver unchanged, or the PSF-weighted super-resolution imaging method developed in Appendix B is used).This will form a super-resolution absorption image of the film showing the two defects at a resolution (a few nanometers) that is similar to that of a scanning electron microscope (SEM) [55] but without needing a vacuum and the expenses of the SEM.If the camera imaging system in Fig. 1(b) is used (see Eqs. ( 28) and (29) in Appendix A.4 below), multiple sparsely-populated quantum dots can be used to speed up the imaging process as is in the ULM [45]- [47] and PALM [27].)As is seen above, in a practical implementation of the PSFmodulation method, the modulator can be a physical substance such as a small copper bead (Fig. 3(a)), a microbubble, a magnetic particle, an electrically-changed particle, a nanoparticle (or nanodroplet), a gas vesicle, a quantum dot, and localized fluorescent molecules in STED [22], or it can be created remotely as in Fig. 9 and Appendix A.2 below.Also, in addition to moving the modulator directly by motors (Figs. 3,  5, 6, and 7), the physical particles can be manipulated remotely using techniques based on radiation forces [73] such as acoustical [74] and optical [79] tweezers, or based on electrical [64][80]- [81] and magnetic [63][82] forces.
To increase the SNR for super-resolution imaging, modulators (see ( ) m r in Eqs. ( 6) and ( 10)) of patterns that cover a larger area and contain stronger high spatial frequency components than those of the point modulators given in Eqs. ( 13) and ( 19) can be used.For example, the patterns can be 1D sinusoidal chirp strips or a 2D or 3D shape of a broad spatial bandwidth.Such patterns can be produced remotely by radiation forces or can simply be made by embedding a desired shape in a piece of material.Using such patterns as modulators, signal processing techniques such as the inverse filtering [83] and Wiener filtering [84] can be more effectively performed to obtain super-resolution images since the increased higher spatial frequency components of the PSF will provide additional information in the frequency ranges that would otherwise only contain noise.However, the major drawbacks of using such patterns are that the image reconstruction may be more complicated and sidelobes in the reconstructed images may be higher.For example, depending on the pattern used, tomographic image reconstruction [60] may be needed to obtain super-resolution images.

2) Phase Modulation with a 1D Array
Below is an example of using a broadband 1D linear or phased array transducer of f = 2.5-MHz center frequency and D = 25-mm aperture for 2D super-resolution B-mode ultrasound imaging in a media of a speed of sound of about c = 1500 m/s.If the transducer is electronically focused at a depth of z = F = 100 mm, where F is the focal length, the pulse-echo FWHM diffraction-limited resolution of the imaging system can be estimated with Eq. ( 22) and is about 2.47 mm.If the transducer is apodized on both edges with a phase that is opposite from that on the center segment of the transducer, two focal spots near the transducer axis will be formed to produce two shear waves that will interfere when they meet on the axis.If we modulate the amplitude of a 2.5-MHz sine wave with a low frequency, say, 1 KHz, square wave for a few cycles (realized by turning on and off of the sinewave transmission), say, 4-10 cycles (each cycle lasts 1 ms), the shear waves produced will have a narrow bandwidth with a center frequency of 1 KHz.If the speed of shear wave in the media is 1 m/s, the wavelength of the shear wave will be 1 mm.This means that the smallest width of the interference pattern of the shear wave around the transducer axis will be about 1/2 of the shear wave wavelength or 0.5 mm.After the interference pattern of the shear wave is formed, a broadband focused plane wave pulse can be transmitted to get a pulse-echo image.In this case, the phase of the imaging wave will be modulated by the shear wave near the axis.Repeating the shear wave generation and pulse-echo imaging sequence above on different spatial positions by electronic beam steering, a 2D super-resolution Bmode image (one dimension is in the beam steering direction and another is in the time or axial direction) with a lateral resolution close to 0.5 mm can be reconstructed after coherently subtracting the phase-modulated image from the conventional 2D B-mode image (without modulation) using Eq. ( 14) (the modulator can be described by Eq. ( 19) except that the modulator is now one dimensional but with a thickness of the depth of field (DOF) of the transducer).Because the DOF of the focused transducer is limited, the height of the 2D super-resolution B-mode image may be small, especially at a small focal distance.Thus, it is necessary to change the focal distance a few times and then montage a few segments of 2D super-resolution B-mode images to form an image of a larger field of view.The final super-resolution image can be superimposed on top of the conventional B-mode image to provide doctors with additional diagnosis information.Increasing the shear wave frequency from 1 KHz to 2 KHz, the image resolution can be doubled.However, as mentioned in Section VI.B, the ultimate limit on image resolution is the SNR of the imaging system.To reduce the sidelobes of the modulator (shear wave interference pattern), an image produced by shear waves of a very high frequency, say, 10 KHz, can be used to subtract coherently from the image produced by 1 or 2 KHz since the high-frequency shear wave contains similar sidelobes but not the interference pattern around the axis (mainlobe) due to a high attenuation.

3) Using Microbubbles or Nanodroplets as Modulators
Microbubbles in the ultrasound localization microscopy (ULM) [8]- [10], [45]- [47] can be used as modulators to implement the PSF-modulation method to reconstruct a superresolution image of blood vessel walls and their surrounding soft tissues using Eq. ( 14) if the microbubbles are immediately above the vessel walls so that ultrasound beam reaches the microbubbles before hitting the vessel walls in a way similar to the experiments in Figs.3-5 (if the microbubbles are immediately below the vessel walls, super-resolution images of a different contrast mechanism may be reconstructed in a way similar to the super-resolution imaging with quantum dots [53][54] above in Appendix A.1).Because the signals from the blood are much smaller than those from the blood vessel walls and soft tissues if the ultrasound frequency used is not too high, say, 15-25 MHz, the area of blood will appear dark in the reconstructed super-resolution image.The ability to get microscopic images of blood vessel walls and surrounding soft tissues deep in the body will help to distinguish benign tumors from malignant without resorting to invasive biopsy or invasive intravascular ultrasound (IVUS) imaging [85] that does not work in capillaries.To implement the method, after localizing the centers of the sparsely-populated microbubbles in each frame of RF B-mode image, the RF A-lines through the centers of the microbubbles are subtracted with those obtained before the microbubble injection (or with those in the nearby image frames in which these microbubbles have moved to elsewhere).The subtraction can be done within a short time window (say, a half period of the imaging wave) in which the isolated microbubbles are present.Tracking the individual microbubbles over multiple frames of the RF B-mode images acquired at a very high image frame rate and using the technique in the last paragraph of Section II "Theoretical Preliminaries" to realign the axial positions of each microbubble or to use the "I" and "Q" components of the RF signals without the need of the realignment, the DC component ( )

PE
C t in Eq. ( 16) can be removed as long as the microbubble and the magnitude of the PSF do not change significantly at the tracked positions of each microbubble (setting a proper timegain-control (TGC) curve will help to minimize the change of the magnitude of the PSF).The subtracted RF signals that represent the center locations of the microbubble can be superposed incoherently in a way similar to that was done in the experiments in Figs.3-7 to get multiple pixels of the final image (see the last paragraph of Section IV "Computer Simulation and Experiment Procedures").Accumulating a large number of pixels processed will result in a superresolution image of the blood vessel walls and their surrounding soft tissues.When tracking a microbubble, if it starts to move out of the plane in the elevation direction of a 1D array transducer, those positions should be ignored.To avoid microbubbles moving out of plane, a 2D array transducer for 3D B-mode imaging can be used.To increase the SNR, the size of the microbbules can be increased to, say, 10 μm or larger, or the ultrasound frequency is increased from, say, 25 MHz.Because ( ) PE C t may be large for microbubbles, the imaging system may need to have a large dynamic range (see Section VI.C "Dynamic Range of the Receiver").
Notice that instead of using the microbubbles as in the ULM above, activated phase-change perfluorocarbon nanodroplets [49] can be used as modulators to obtain nanoscale super-resolution images of the interior of the cells using the PSF-modulation method if the perfluorocarbon nanodroplets activated have a similar size (the activations can be controlled by laser [86] or ultrasound [87]).In addition to pulse-echo imaging, the phase-change perfluorocarbon nanodroplets also can be used as modulators in transmission super-resolution imaging (see the camera in Fig. 1(b)) of the interior of the cells at a nanoscale resolution using Eqs.( 28) or (29) in Appendix A.4 below.The advantage of using transmission imaging is that the received signals due to ( ; ) R C r t in Eq. ( 11) may be much smaller than those of ( ) PE C t in Eq. ( 7).

4) Camera Imaging System
For the camera in Fig. 1(b), the PSF-modulation method also can be implemented to get super-resolution images (notice that Fig. 1(b) can represent many imaging systems such as conventional bright-field optical microscopes, mobile optical microscopes based on cell phones [88], acoustical cameras [61], and so on).Assuming that the camera is an LSI system, the image on the receiver array of the camera can be written as (see Eq. ( 9)): ' ' , ( ; ) PSF ( '; ') ( '; ') ' ' PSF ( ; ) ( ; ) where ( ; ) f r t is an object function that can represent the spatial wave field on the object plane at any given time, PSF ( ; ) α R c r t is the PSF of the camera, the subscript "c" in Eq. ( 24) means "camera", 1/ 0 α σ = > is a real number, σ is the magnification of the imaging system.Multiplying the object function with a modulator ( " )    27) is the same as Eq.(10) except that it has a constant spatial scaling factor α and ( ) r is a part of the modulator as explained in the texts below Eq. (11).Subtracting Eq. ( 27) from Eq. ( 24), super-resolution images can be reconstructed using a modulator that is similar to the one given in either Eq.(13) Eq. ( 28) can be implemented as follows.Taking two images with the camera, one with (see Eq. ( 25)) and another without (see Eq. ( 24)) the modulator ( " ) r , which has a small size compared to the PSF and is placed in the object plane at position " ( ", ") = r x y .Then, subtract the first image from the second to get a pixel (at α r ) of the super-resolution image after setting " ( , ) = = r r x y (see Eq. ( 28)) and incoherently superposing the resulting temporal signal over about a half period of the imaging wave.To increase the SNR, instead of setting " = r r , the subtracted image at a given " r , , ( ; "; ) can be integrated over α r that represents the aperture of the receiver array.Moving the modulator over the object in the object plane by changing " r and repeating the process above, a super-resolution image that is a filtered version of ( ; ) f r t (due to the time convolution * t with the PSF) can be reconstructed.If the modulator changes its position in the z (axial) direction while moving, Eqs.(28) or (29) can still be used as long as the magnitude of the PSF does not change significantly (see the explanations in the last paragraph of Section II "Theoretical Preliminaries").To increase the imaging speed, as is in the ULM [8]- [10] [49] or PALM [27], multiple randomly-distributed modulators can be used to get multiple pixel values of the super-resolution image at once as long as the diffraction-limited images of these modulators do not overlap with each other on the receiver aperture and the modulators are identical (if not, the image contrast will include the variations due to the modulators unless such variations can be compensated).Accumulating multiple frames of such images, the final super-resolution image can be obtained (if more values are obtained at a pixel, an average of the values can be used to increase the SNR).Notice that if the receiver array also is used as a transmitter to illuminate the object with a plane wave, the camera can be viewed as a pulse-echo imaging system where Eqs. ( 4) and ( 6), instead of Eqs. ( 9) and ( 10) (or Eqs. ( 24), (25), and ( 27)), can be used to reconstruct superresolution images.
An interesting application of using multiple modulators with the camera in Fig. 1(b) is to image objects such as proteins with an apparatus that is similar to a scaled-down electrophoresis [64].Embedding the objects in a thin gel or a scaffold that is placed in the object plane within the depth of focus of the camera and letting the objects be illuminated from the opposite side of the receiver array, a nanoscale superresolution image of the objects can be reconstructed using Eqs.( 28) or ( 29) by taking multiple images at a high frame rate while sparsely-distributed, uniformly-sized, and opaque nanoparticles (used as modulators) are moving randomly through the gel or scaffold under an electrical (for charged nanoparticles) or magnetic (for magnetic nanoparticles [63]) force.
The advantages of using the camera in Fig. 1(b) as compared to the imaging system in Fig. 1(a) is that the diffraction-limited image of the object can be viewed directly, which helps to locate the region of interest for super-resolution imaging.Also, using the localization technique for microbubbles (in ULM) or fluorescent light (in PALM), the center positions of the diffraction-limited images of the modulators (such as nanoparticles) on the receiver array can be accurately determined with the camera.In addition, using the camera, there is no need to scan point-by-point a focused beam along with the modulator to form an image, simplifying the imaging system.

5) Optical Imaging Systems
In an optical imaging system, intensity of the waves is usually measured.Thus, in some implementations where coherent processing (summation or subtraction) is needed, the phase of the optical signals needs to be recovered by techniques such as holographic [51]- [52][89], iterative [90], and neuronetwork [91] methods for the PSF-modulation method to work.However, in an optical protein (or other substance) super-resolution imaging using the camera in Fig. 1(b) and multiple sparsely-populated moving modulators mentioned above (the sparsity of the modulators is to ensure that each modulator can be treated individually as in the ULM [45]- [47] or PALM [27]), a continuous coherent subtraction can be accomplished by an interferometer method to get images in Eq. ( 28) or (29).In the subtraction process, the modulators move and the delay time between the two arms of the optical signals is adjusted so that there is no overlap of the diffraction-limited images of each modulator before and after it moves and thus the center or the averaged value in the area of the subtracted image of the modulator can be determined.Accumulating multiple frames of images in which the centers or the averaged values in the areas of the subtracted images of the modulators are obtained, super-resolution images can be reconstructed.Because of a limited coherent length of the optical wave, the speed of the modulators must be high and thus a high-speed camera is needed to snap the subtracted images.Due to the use of a high-speed camera and fast-moving modulators, dynamics of the objects (such as protein dynamics) can be studied at both high temporal and spatial resolutions.
Although in principle, one needs to know the phase of optical signals for a coherent subtraction, in some cases, the coherent subtraction to reconstruct super-resolution images can be implemented approximately using the light intensity directly.Let's assume that the light signals  .The phase modulation to reconstruct super-resolution images is similar to that given in Figs. 8 and 9, and in the example of Appendix A.2 "Phase Modulation with a 1D Array".Notice that the phase modulation can be accomplished by using either a physical particle or shear wave as a modulator.If Other applications in optics include the wide-field lensless in-line digital holographic imaging system [51]- [52].In such a system, a narrowband point light source from a far distance (a few centimeters) is used to illuminate a thin layer of object such as cells grown on a thin glass petri dish and a Charge-Coupled Device (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) sensor array that is placed in parallel with and at about a few hundreds of micrometers behind the object plane is used to record the interference pattern.Both amplitude and phase images of the object can be reconstructed approximately by backpropagating the angular spectrum [1] of the recorded holographic interference pattern to the object plane under the assumption that the waves scattered from the object is much weaker than the incident wave [92].If nanoparticles such as magnetic nanoparticles [63] are introduced to the fluid media where the object such as live cells are located, super-resolution images of the cells can be reconstructed at a nanoscale resolution in a way similar to obtaining super-resolution images using the camera in Fig. 1(b) above.The magnetic nanoparticles can be moved randomly by a changing force of an external magnetic or electromagnetic field, and the CCD or CMOS receiver array is used to take pictures of the resulting interference patterns continuously for image reconstruction.(Notice that replacing the optical wave with ultrasound and choosing appropriate parameters, superresolution in-line acoustical holographic images can be reconstructed using a narrowband plane-wave illumination.Unlike optics, both the amplitude and phase of the waves on the receiver array can be obtained in ultrasound and thus images can be reconstructed without the twin-image artifacts [92] by backpropagating the angular spectrum [1] of the received signals to the object plane.)Another potential application of the PSF-modulation method is optical coherence tomography (OCT) [56] that is similar to an ultrasound B-mode imaging system to get superresolution images with a deeper penetration if the phase of the signals can be obtained and proper modulators can be found.Also, if a fluorescent light could be used as a modulator, nanoscale imaging would be possible by directly using either STED [22] or PALM [27].

6) Other LSI Imaging Systems
Since, in principle, Eqs. ( 4), ( 6), (9), and (10) work for any LSI system to get super-resolution images, the PSF-modulation method should work for various imaging systems such as magnetic resonance imaging (MRI) [57] if the modulated PSF has a higher spatial frequency.Also, for nondesctructive evaluation (NDE) of materials and other areas of science and engineering, the PSF-modulation method can be used for super-resolution imaging on some surfaces that are difficult to be inspected by sensors or near-field imaging techniques [17][34]- [35] due to conditions such as high temperature, limited space access, and hazardous environment that may damage sensors.As mentioned before, to speed up imaging, multiple modulators can be used as long as their diffractionlimited images do not overlap and the modulators are identical.
The PSF-modulation method also can be applied to scanning acoustical microscopy (SAM) [58][59] or other similar imaging systems for super-resolution imaging (see Fig. 1(a)).In such imaging systems, a modulator (such as a microbubble, a small particle, a particle that by itself emits waves [22][27][78] [54], and a small hole or particle embedded in a thin sheet of material that is larger than the focal spot of the transducer) is fixed at the center of the focal area of the transducer and an object is scanned in the object plane as in Figs. 3, 5-8.The transducer can be used either in a pulse-echo or receiver-only mode (in the receive-only mode, the object can be illuminated from the opposite side of the transducer).Also, the modulator can be placed either on the top or bottom surface of the object (or even inside the object).To fix the position of the modulator relative to the transducer, the modulator can be held in position by an external force, confined in a microchannel, or embedded in a thin sheet of material.If a microchannel is used, it can be placed with the object while the modulator moves through the microchannel to keep the relative position of the modulator and the transducer unchanged to reconstruct a line of super-resolution image.Translating the microchannel in the direction that is perpendicular to the microchannel, super-resolution images can be obtained line by line.If multiple parallel microchannels are used, the translation of the microchannel is not necessary.If a modulator embedded in a thin sheet of wave-absorbing material is used, where the modulator can simply be a small hole or another material that allows the waves to penetrate (or to reflect/scatter) to implement the modulation function '( ) 1 ( ) = − m r m r (see Eqs. ( 13), ( 15), ( 18), ( 19), (20), and ( 21)), the object can be slid under (when the modulator is transparent) or above (when the modulator is a reflector/scatterer) the sheet of the material to reconstruct super-resolution images in the pulse-echo imaging system without needing a subtraction (the object can be a thin slice of specimen as in Refs.[58][59]).In a transmission imaging system, if the modulator is a small hole on a thin sheet of wave-absorbing/blocking material, or if a wave-emitting modulator [22][27] [78][54] is used, super-resolution images also can be reconstructed without subtraction (see Appendix A.1 above).If the camera system in Fig. 1(b) is used [61], multiple sparsely-populated modulators can be used to speed up the image reconstruction as mentioned before (see Appendix A.4).The major advantages of super-resolution imaging with '( ) m r above are that the dynamic range of the imaging system does not have to be very large (see Section VI.C) and the motion artifacts can be reduced (see Section VI.F).
In terms of the wave source/filed imaging such as the photoacoustic imaging [60], it is possible to reconstruct superresolution images using the PSF-modulation method by injecting small physical modulators into the body or producing shear waves remotely for phase modulation.Notice that the injected modulators can change both the absorption of the light near the locations of the modulators and the ultrasound waves produced by the heating of the light to contribute to the contrast of the reconstructed super-resolution images.
In underwater acoustics [61], fishes or other substances moving randomly could also be used as modulators to get super-resolution images of the seabed/riverbed terrain (twoway), the sinking ships/objects (two-way), noise pattern emitted from submarines/underwater vehicles (one-way), and other underwater objects if the moving fishes or other substances are sparsely populated and are close to the objects or the wave source/field to be imaged.Since the sizes of the fishes may be different and the shape of each fish may change in the direction of the impinging waves, compensation for such changes is needed for the reconstruction of super-resolution images.

B.
PSF-Weighted Super-Resolution Imaging In this subsection, the theory developed in Section II "Theoretical Preliminaries" is extended to simplify the imaging system by keeping both the transducer and object to be imaged stationary during imaging.
Assuming in Fig. 1(a) that both the transducer and the object ( ) f r to be imaged are stationary in space and the PSF of an LSI and LTI pulse-echo imaging system is PSF ( ; ) PE r t , the signal received by the transducer is given by (see Eq. ( 4)): where the subscript "w" means "weighted".Adding a modulator ( ) m r and then moving it in the space, an image can be formed (see Eq. ( 6)): γ PE m r was explained in the texts under Eq.( 7).Subtracting Eq. ( 31) from Eq. ( 30), we have (see Eq. ( 14)): From Eq. (33), 4D (3D in space plus 1D in time) PSF-weighted super-resolution images of the function PSF ( ; ) ( ) PE r t f r at any given time t can be reconstructed if the maximum spatial frequency of the function [1  ( )] − m r is higher than that of the point spread function PSF ( ; ) PE r t and ( , ) PE w C r t is a constant at any given time t .In a plane-wave pulse-echo imaging system, PSF ( ; ) PE r t is a constant at a given distance z and time t (assuming that the plane wave propagates in the z direction), PSF ( ; ) ( ) PE r t f r = ( ) f r and thus a superresolution C-mode image of ( ) f r can be reconstructed.In fact, the wave produced by a focused disc transducer at its focal distance can be approximated with a plane wave near the z axis and thus ( ) f r can be reconstructed if the image field of view is small compared to the focal size.In the case of plane wave, ( , ) PE w C r t is a constant at any given time t and thus it can be removed from the reconstructed images.If the modulator also moves in the z direction while getting the Cmode image, such motion can be compensated using the method given in the last paragraph of Section II "Theoretical Preliminaries".
Similarly, assuming that ( ; ) f r t represents a spatial field of a wave source at any given time t , 4D PSF-weighted superresolution images of the wave field can be reconstructed using the signals received by a receiver (one-way imaging).Keeping the wave source and the receiver stationary in space, the received signal is given by (see Eq. ( 9)): γ R m r is a part of ( ) m r , as was explained in the texts under Eq.( 11).Subtracting Eq. ( 35) from Eq. ( 34), we have (see Eq. ( 17)): Similar to the case of pulse-echo imaging system, PSF ( ; ) ( ; ) * R t r t f r t at any given time t can be reconstructed.Using a receiver of a plane-wave response (or a focused wave in the focal area), at a given distance z , PSF ( ; ) R r t will only be a function of time t when the plane wave vector is in the z direction.
In z t f r t can be reconstructed at a fixed z and time t , which is a filtered version of ( ; ) f r t .Also, using a modulator that is a wave emitter such as the quantum dot [53][54] as in the third paragraph of Appendix A.1 above, PSFweighted super-resolution imaging at a nanoscale resolution (a few nanometers) is possible without a subtraction.If the modulator changes its position in the z direction during the Cmode imaging, Eq. ( 37) can still be used as long as the magnitude of the PSF does not change significantly (see the explanations in the last paragraph of Section II "Theoretical Preliminaries").
Notice that for a phase modulation, both ( ; ) PE w C r t and ( , ) R w C r t are zeroes as was explained in the texts below Eq. (21).Also, compared to the PSF-modulation method (except for the camera in Fig. 1(b) and Appendix A.4), the major advantages of the PSF-weighted super-resolution imaging are that only the modulator needs to be moved in space (or remotely produced one by one) and only one reference signal in either Eq.(30) or Eq. ( 34) is needed, which simplifies the imaging system.The disadvantages are that the reconstructed images may be weighted by the PSF as in Eqs. ( 33) and ( 37), it is difficult to use multiple modulators moving randomly to speed up the imaging speed (unlike the camera in Fig. 1(b) and Appendix A.4, the signals from the multiple modulators will be mixed), and the field of view of the images may be limited when a focused wave is used or the SNR is lowered when a plane wave is used.
respectively, and * k represents a convolution with respect to k .Since ( ) PE C t is not a function of position r , its spatial Fourier transform is a delta function, i.e., ( ) PE C t is a DC (direct current) component of the image ( ; ) m PE R r t at any given time t .Because the convolution in Eq. (8) is performed in the spatial frequency domain k , the maximum spatial frequency of the modulated PSF in Eq. (6), PSF ( ; ) PSF ( ; ) ( ) = m PE PE

Fig. 1 .
Fig. 1.Imaging systems with modulation of the point spread function (PSF).(a) Modulation of the PSF of an imaging system that uses a spherically focused wave (such as ultrasound, optical, and electromagnetic waves) for four-dimensional (4D) super-resolution 25 mm; temporal frequency f = 1 MHz (continuous wave, i.e., CW); the speed of sound c = 1500 m/s; wavelength λ = 1.5 mm; transducer diameter D = 25.4 mm with a focal length of F = 50 mm; number − f = / F D = 1.97; phase shift 0 φ = /10 π ; and infinite SNR (i.e., no noise).
Fig. 3(e) is a photo of a physical point object and the 1-MHz center frequency and 25.4-mm diameter focused broadband transducer mounted in a copper tube of 38.1-mm inner diameter.The point object was made of a 0.25-mm diameter and 6-mm long stainless steel wire mounted on a corn-shaped wax rod.An acoustical lens made of acrylic was attached to the front surface of the transducer and the modulator was placed in the focal area of the transducer.Figs.3(f), 3(g), and 3(h) are experiment results corresponding to Figs. 3(b), 3(c), and 3(d) respectively.The distance between the modulator and the point object was about 0.5 mm in the experiments.

Fig. 3 .
Fig. 3. Two-way (pulse-echo) C-mode imaging of a point object.(a) A photo of the modulator (a piece of copper wire of 0.5 mm in diameter and about 1 mm in length) glued on the cross of two perpendicular copper wires (about 85 μm in diameter) that were mounted on a copper ring of 38.1-mm inner diameter.(b) and (c) are simulated images of the point object with and without the modulator respectively.In the simulations, the point object was a geometric point without a physical dimension, the transducer was driven by a 1-MHz continuous wave (CW) signal, and the modulator was an infinitely-thin 0.5-mm diameter disc at the center of the focal plane of the transducer and had a 100% amplitude modulation.Other parameters of the simulations are in the figure.(d) Super-resolution image reconstructed from (b) and (c).(e) A photo of the 1-MHz transducer focused with an acoustic lens, the modulator, and a point object.(f), (g), and (h) are images corresponding to (b), (c), and (d) respectively and were obtained using the experiment system in Fig. 2. In the experiments, the point object was a stainless steel wire of about 250 μm in diameter and 6 mm in length, and the distance between the modulator and the point object was about 0.5 mm.The grayscale bar represents the normalized magnitude of sound pressure received.

Fig. 4 .
Fig. 4. Horizontal line plots through the center of the images in Fig. 3.The solid (black) and the dotted (blue) lines are through the center of the experimentally-obtained images in Figs.3(h) (super-resolution) and 3(g) (diffraction-limited) respectively.The dash-dotted (red) and the dashed (pink) lines are through the center of the simulated images in Figs.3(d) (super-resolution) and 3(c) (diffraction-limited) respectively.

Figs. 5
Fig.5shows pulse-echo imaging of an "L"-shaped object that was consisted of 11 point objects.Fig.5(a)shows the dimensions of the object.On the left in the vertical direction, there were 6 point objects with spacing of 2.5, 2.0, 1.5, 1.0, and 0.5 mm respectively from top to bottom.In the horizontal direction, there were 5 point objects with spacing of 0.5, 1.0, 1.5, and 2.0 mm from left to right respectively.The vertical distance between the right-most two point objects was 0.5 mm.Figs.5(b)-5(d) are the same as Figs.3(b)-3(d) respectively, except that the "L"-shaped object in Fig. 5(a) was imaged (the object consisted of geometrical points without a physical dimension).Fig. 5(e) is a photo of a physical "L"-shaped object that was mounted on a clamp and was consisted of 11 stainless wires of about 0.25 mm in diameter and 6 mm in length.The wires were molded onto an epoxy resin base.Figs.5(f)-5(h) are the same as Figs.3(f)-3(h) respectively, except that the object in Fig. 5(e) was imaged.From both 5(d) and 5(h), the point objects that were separated by 0.5 mm in the "L"-shaped object are visually identifiable (barely in Fig. 5(h)), although the diffraction-limited resolution without the modulator was 2.65 mm in the experiment.

Fig. 5 .
Fig. 5.This figure is the same as Fig. 3 except that an L-shaped object consisting of 11 points were imaged.(a) Dimensions of the Lshaped object.(b)-(d) Simulated images corresponding to Figs. 3(b)-3(d) resppectively.(e) A photo of the L-shaped object used in the experiments.(f)-(h) Experiment results corresponding to (b)-(d) respectively.

Fig. 6
Fig. 6 shows one-way C-mode images of the wave source/field produced experimentally by a 48-element linear array transducer.Fig. 6(a) is a photo of the linear array transducer mounted on a clamp.Fig. 6(b) gives the dimensions of the array.Fig. 6(c) is a close-up photo of the front surface of the array (the array does not have a front matching layer).The array was driven by the 1-cycle sine wave (pulse) of 1 MHz (see Fig. 2) via two home-made RF power amplifiers of opposite polarities (180° phase difference).The polarities of the drive signals changed every two elements across the entire array, i.e., + + − − + + − − ⋅⋅⋅ + + − − + + − − .The waves produced by the array were received by the 1-MHz focused transducer.Figs.6(d) and 6(e) are C-mode images obtained

FWHM
the parameters of the experiments ( λ = 1.4826 mm, F = 44.12mm, and D = 25.4 mm), the resolution calculated with Eq. (23) is about 3.63 mm.Since the pitch of the array in Fig.6(b) was 0.381 mm and every two elements (0.762 mm) were driven with the same polarity, from Fig.6(f) it is clear that the image resolution is at least 0.762 mm.The darker area on the left in Fig.6(f) may suggest that the array was partially damaged.

Fig. 6 .
Fig. 6.Experimentally obtained one-way C-mode images of a wave source/field produced by a 48-element ultrasound linear array transducer that was driven by the 1-cycle sine wave (pulse) of 1 MHz in alternating phases (180° phase difference) for every two elements (i.e., ++--++--… ++--++--).The modulator was about 0.5 mm away from the surface of the array and the array does not have a front matching layer.The spherically focused transducer in Fig.2was used as a receiver.(a) A photo of the linear array (indicated by the arrow) mounted on a clamp.(b) Dimensions of the array.(c) A photo of the front surface of the array.(d) and (e) are C-mode images of the ultrasound wave produced by the array with and without the modulator respectively.(f) Super-resolution image reconstructed from (d) and (e).The grayscale bar represents the normalized magnitude of sound pressure.

Fig. 7
Fig. 7 is the same as Fig. 6 except that the wave source/field was produced by a patterned 25.4-mm diameter disk transducer.The transducer was air backed and enclosed in a plastic tube that was wrapped by a copper sheet for electrical shielding.The patterned transducer was driven by the 1-cycle sine wave (pulse) of 1 MHz and the waves produced were received by the 1-MHz focused transducer.Fig. 7(a) is a photo of the patterned transducer mounted on a clamp.Fig. 7(b) is the dimension of the pattern of the back electrode.The outer ring of the back electrode was connected to the front electrode that was grounded (GND), which gave an effective transducer diameter of 22.2 mm.Fig. 7(c) is a photo of the pattern of the front electrode.The pattern contained both horizontal and

Fig. 7 .
Fig. 7.This figure is the same as Fig. 6 except that a patterned transducer was used.(a) A photo of the patterned transducer mounted on a clamp.(b) Dimensions of the pattern of the back electrode of the transducer.(c) Dimensions of the pattern of the front electrode.The outer ring in (b) and the front electrode were connected to the ground (GND), which reduced the effective aperture of the transducer to 22.2 mm.The line spacing (between inner edges of adjacent lines) of the pattern in (c) in both horizontal and vertical groups were 4, 2, 1, 0.5, and 0.25 mm respectively, and the line width of each line was 0.25 mm.(d) and (e) are images obtained with and without the modulator, and (f) is the super-resolution image reconstructed from (d) and (e).The lines separated by 0.5 mm can be clearly seen in (f), suggesting that the resolution was about 0.5 mm.

=
and a = 0.25 mm.Other conditions for the simulations were the same as those in Figs.3 and 5. Figs.8(a) and 8(b) are simulated pulse-echo C-mode images of a point object (a geometric point) with and without the modulator respectively.The real (left) and imaginary (right) parts of the images in Figs.8(a) and 8(b) are shown as small insets near the bottom.Fig. 8(c) is the super-resolution image reconstructed with Eqs.(14) and (20).Figs.8(d)-8(f) are the same as Figs.

Fig. 8 .
Fig. 8.This figure is the same as the simulation portions of Figs. 3 and 5 except that the phase modulation in Eq. (19) was used, where 0 φ = /10 π and a = 0.25 mm.(a)-(c) Images corresponding to those in Figs.3(b)-3(d) respectively.The small image insets in both (a) and (b) are real (left) and imaginary (right) parts of the images.(d)-(f) Images corresponding to those in Figs.5(b)-5(d) respectively.
a spatial position are received respectively by a point detector before and after introducing a modulator that modifies N s to m s (see Fig.1), where , i.e., the modulator may cause both amplitude and used to reconstruct super-resolution images.Again, multiple sparsely populated modulators can be used to speed up the image reconstruction if all modulators have a similar size, shape, opacity 0 m , and phase 0 φ .

'
or Eq.(19): and then moving it in space, an image is obtained as follows (see Eq. (10)):