Multi-Volumetric Refocusing of Light Fields

Geometric information of scenes available with four-dimensional (4-D) light fields (LFs) paves the way for post-capture refocusing. Light field refocusing methods proposed so far have been limited to a single planar or a volumetric region of a scene. In this letter, we demonstrate simultaneous refocusing of multiple volumetric regions in LFs. To this end, we employ a 4-D sparse finite-extent impulse response (FIR) filter consisting of multiple hyperfan-shaped passbands. We design the 4-D sparse FIR filter as an optimal filter in the least-squares sense. Experimental results confirm that the proposed filter provides 63% average reduction in computational complexity with negligible degradation in the fidelity of multi-volumetric refocused LFs compared to a 4-D nonsparse FIR filter.

. Refocusing of the "Bush" LF; (a) refocused for a single volumetric region [21]; (b) refocused for two volumetric regions using the proposed 4-D sparse FIR filter. 4-D linear filter having a hyperfan-shaped passband to achieve volumetric refocusing. Dansereau et al. [19] and Premaratne et al. [20] employed similar 4-D linear hyperfan filters for LF denoising. In [21], Premaratne et al. proposed a 4-D sparse FIR filter having a hyperfan-shaped passband, designed using the windowing technique and hard thresholding, for volumetric refocusing. This sparse filter provides significant reduction in computational complexity with negligible degradation in the fidelity of refocused LFs compared to that proposed in [18]. Recently, Wang et al. [22] employed depth-based anisotropic filter and superresolution approach for LF refocusing. Furthermore, Yang et al. [23] and Pei et al. [24] proposed spatial-domain optimization techniques for all-in-focus refocusing methods, mainly for attenuating occlusions. The computational complexity of these methods are considerably high compared to 4-D-FIR-filter based methods. All of these prior works are predominantly limited to LF refocusing over a single planar or a single volumetric region.
In this letter, we demonstrate simultaneous refocusing of multiple volumetric regions in LFs albeit at lower computational complexity. To this end, we employ a 4-D sparse FIR filter having multiple hyperfan-shaped passbands. Fig. 1 shows refocusing of a single volumetric region obtained with the 4-D sparse FIR filter proposed in [21] and simultaneous refocusing of two volumetric regions using the proposed 4-D sparse FIR filter. It is evident that multi-volumetric refocusing can emphasize multiple objects or regions occupying different depth-ranges in an LF simultaneously. This feature may open new avenues, in particular, in LF photography [15] and cinematography [25]. In addition, our method can achieve refocusing of a single volumetric or a planar region as well because these are special cases of multi-volumetric refocusing. We design the 4-D sparse FIR filter by employing a two-step sparse filter design method proposed in [26], which considers the design of filters having quadrantally-symmetric impulse responses. Our 4-D sparse FIR filter has a centro-symmetric impulse response, and we adapt the two-step method in [26] appropriately to design our filter. Our 4-D sparse FIR filter is optimal in the least-squares sense. Experimental results obtained with the LFs in the EPFL data set [27], Stanford dataset [28] and the 4-D benchmark dataset [29] confirm that the proposed 4-D sparse FIR filter provides 63% average reduction in computational complexity with negligible degradation in the fidelity of two-volumetric-region refocused LFs compared to an equivalent 4-D non-sparse FIR filter. Furthermore, our sparse filter provides 13% reduction in computational complexity compared to that proposed in [21] with similar fidelity for refocusing of a single volumetric region.

II. REVIEW OF THE SPECTRUM OF A LIGHT FIELD
The spectrum of a LF is briefly reviewed in this section. To this end, we first consider standard two-plane parameterization, with the globally defined image-plane coordinates, of a Lambertian point source as shown in Fig. 2. Note that (n x , n y ) ∈ Z 2 and (n u , n v ) ∈ Z 2 denote the 2-D discrete-domain camera-plane and image-plane coordinates, respectively, and D is the constant distance between the camera and image planes. The 4-D LF l p (n), where n = (n x , n y , n u , n v ) ∈ Z 4 , corresponding to the Lambertian point source located at (x 0 , y 0 , z 0 ) ∈ R 2 × R + and having an intensity l 0 can be modeled as [30], [31] where m = D z 0 − 1 and Δ i , i = x, y, u, v is the sampling interval along the dimension i. Note that the LF consists of a plane having a constant value l 0 given by the intersection two hyperplanes P xu and P yv . In this case, the region of support (ROS) R p of the spectrum L p (ω), where ω = (ω x , ω y , ω u , ω v ) ∈ R 4 , inside the principal Nyquist hypercube N ( {ω ∈ R 4 | − π ≤ ω i < π, i = x, y, u, v}) is given by R p = H xu ∩ H yv [30], [31], where The ROS R p is a plane through the origin of ω inside N , of which the orientation depends only on the depth z 0 of the Lambertian point source. Note that we do not consider the finite sizes of the camera and image planes in presenting the spectral ROS for simplicity. However, even with these constraints, the spectral ROS predominantly occupies the region defined by the ROS R p [32], [33].
In order to obtain the ROS R o of the spectrum of a LF corresponding to a Lambertian object, we can consider the Lambertian object as collection of Lambertian point sources located in a volumetric region with a depth range [30], and corresponds to a hyperfan inside N [18] (shown in Fig. 3(a) in the ω x ω u subspace). In the case of multiple objects located in M volumetric regions, the spectral ROS R M o can be obtained as The spectral ROS R M o thus contains M hyperfans inside N . Therefore, we can refocus M volumetric regions in a LF simultaneously by employing a 4-D filter having M hyperfan-shaped passbands inside N .
This structure is motivated by the partial separability of the spectral ROS of a Lambertian point source and leads to an extremely low respectively. Note that the passband of B of the 4-D FIR filter H(z) given by B xu ∩ B yv completely encompasses the spectral ROS given by (2). Fig. 3(b) shows the passband of H xu (z x , z u ) for M = 2. Our 4-D FIR filter H(z) sharpens (i.e., focuses) the depth ranges in an LF corresponding to these M hyperfans and blurs other depth ranges corresponding to the stopband. Fig. 3(b) shows the parameters that specify the ith hyperfan of the passband of H xu (z x , z u ). Here, α i , θ i and B i determine the orientation, angular-width and the length of the ith hyperfan, respectively, and T i determines the width of the guard band employed to achieve an improved accuracy near the origin of ω [34]. The passband and the specifications of the ith hyperfan is the same for H yv (z y , z v ) in the ω y ω v subspace. Therefore, the design of H xu (z x , z u ) and H yv (z y , z v ) reduces to a single 2-D FIR filter design. We next present the design of H xu (z x , z u ) in detail.
We adapt the 2-D sparse FIR filter design method proposed in [26] to design H xu (z x , z u ). To this end, we express the We design H xu (z x , z u ) as a zero-phase filter [35, ch.3]. In this case, the impulse response of the filter is centro-symmetric, i.e., h xu (n x , n u ) = h xu (−n x , −n u ). Therefore, we can simplify H xu (e jω x , e jω u ) considering the centro-symmetric property as H xu (e jω x , e jω u ) = Here T . Lu and Hinamoto proposed two-step weighted least-square approach in [26] to design 2-D sparse FIR filter having quadrantally-symmetric impulse responses. With (3), we can use this approach to design H xu (z x , z u ) despite its centrosymmetric impulse response. In the first step, we obtain an intermediate sparse impulse response in the least-squares sense. We express the objective function J(h xu ) to be minimized as where H I xu (e jω x , e jω u ) is the ideal frequency response of H xu (z x , z u ) having 1 in the passband and 0 in the stopband, H xu (e jω x , e jω u ) is given in (3), W (ω x , ω u ) is a weighting function that we use to control the stopband attenuation, μ is a small positive number (typically between 0.01 and 1), and F is the region corresponding to the passband and stopband, i.e. without the transition band [26]. Because, H xu (e jω x , e jω u ) is centro-symmetric, we consider only the region [−π, π] × [0, π] in the 2-D frequency domain (ω x , ω u ) to define F. By considering finite set of frequency grid points in F and introducing upper bounds for the first and second terms in the right hand side of (4), we can convert the optimization problem as an 1 -2 minimization problem, which can be converted as a second-order cone programming problem [26]. Due to the limited space, we do not present the detailed steps, and the reader is referred to [26]. The solution h i xu of the second-order cone programming problem is an approximately sparse impulse response [26], and we employ hard thresholding in order to obtain a sparse impulse response h i,s xu , i.e., where th (∈ [10 −4 , 10 −2 ], typically) is the threshold value.
In the second step, we again optimize h i,s xu in the least-squares sense in order to further improve the accuracy. We express this optimization problem as where I ∞ is the set containing indices i for which h i,s xu (i) = 0. This optimization problem is a quadratic program. We obtain the sparse impulse response h s xu as h s xu = γ h q,s xu , where h q,s xu is the solution of (6), and γ (∈ [1, 1.5], typically) is a constant used to compensate the intensity reduction of a refocused LF due to the small number of SAIs.

IV. EXPERIMENTAL RESULTS
We employ three LF datasets, the EPFL data set [27], Stanford dataset [28] and the 4-D benchmark dataset [29], to evaluate the performance of the proposed 4-D sparse FIR filter. We present the experimental results obtained for LFs in the EPFL dataset for M = 2 in this section and those obtained for the LFs in the Stanford dataset and 4-D benchmark dataset for M = 2 and in the EPFL dataset for M = 3 in supplementary results. 1 Furthermore, we compare the performance of the proposed filter compared to a 4-D nonsparse FIR filter in multi-volumetric refocusing. Next, we compare the performance and computational complexity of the proposed filter with those of [18] and [21].

A. Performance of the Proposed 4-D Sparse FIR Filter in Multi-Volumetric Refocusing
We process five LFs, "Parc du Luxembourg," "Bush," "Books," "Sphynx," and "University" in the EPFL dataset using the proposed 4-D sparse FIR filter and a 4-D nonsparse FIR filter. Here, we select only the middle 11 × 11 SAIs for each LF and discard SAIs affected by vignetting. For the "Parc du Luxembourg" LF, we design H xu (z x , z u ) and H yv (z y , z v ) with α 1 = 50 • and α 2 = 120 • , θ 1 , θ 2 = 10 • , B 1 , B 2 = 0.9π rad/sample, T 1 , T 2 = 0.08π rad/sample, μ = 0.1, th = 0.004, γ = 1.4, and W (ω x , ω u ) = 1 for the passband and W (ω x , ω u ) = 2 for the stopband. We present the specifications of H xu (z x , z u ) or H yv (z y , z v ) employed for the other four LFs in the supplementary results 2 . We select the order of H xu (z x , z u ) and H yv (z y , z v ) as 10 × 40 for all the five cases. Note that the order of the resulting 4-D sparse FIR filter H(z) is 10 × 10 × 40 × 40. We employ the CVX [36], [37] optimization toolbox to obtain the sparse impulse responses. The sparse impulse responses of  H xu (z x , z u ) and H yv (z y , z v ) designed for the "Parc du Luxembourg" LF have 153 nonzero coefficients whereas an equivalent 2-D nonsparse FIR filter, designed with th = 0 in (5), has 451 nonzero coefficients. Consequently, the proposed 4-D sparse FIR filter provides approximately 66% reduction in computational complexity compared to an equivalent 4-D nonsparse FIR filter. Fig. 4 shows the magnitude responses of H xu (z x , z u ) for the sparse and nonsparse cases, and the normalized root mean square error between the two frequency responses 4.41%. For all the considered LFs in the three datasets, the proposed 4-D sparse FIR filter provides 63% average reduction of computational complexity with an average normalized root mean square error of 3.68% compared to nonsparse counterparts. This indicates the frequency response of the sparse filter is approximately equal to that of the nonsparse filter despite having considerably less coefficients. Fig. 5 shows the central sub-aperture image (SAI) obtained with the 4-D sparse and nonsparse FIR filters for the "Parc du Luxembourg" LF. The structure similarity (SSIM) index [38] between the central SAIs of the two refocused LFs is 0.9900. Furthermore, the blind/referenceless image spatial quality evaluator (BRISQUE) score [39] of the refocused SAIs corresponding to the 4-D sparse and nonsparse FIR filters are 45.03 and 47.75, respectively. We present the refocused central SAIs, BRISQUE scores and the SSIM indices for the other four LFs in the supplementary results. 1 The average SSIM index between the two refocused LFs is 0.9854, and the average BRISQUE scores corresponding to 4-D sparse and nonsparse FIR filters are 43.95 and 44.97, respectively. These results verify that the proposed 4-D sparse FIR filter provides negligible degradation in fidelity in multi-volumetric refocusing compared to a nonsparse counterpart.

B. Comparison of the Proposed 4-D Sparse FIR Filter With Single-Volumetric-Region Refocusing Filters
Refocusing of a single volumetric region is a special case of the proposed multi-volumetric refocusing. In this subsection, we compare the performance of the proposed filter with those proposed in [18] and [21]. To this end, we consider refocusing of LFs employed in [21]. We design the proposed 4-D sparse FIR filter with α 1 = 50 • , θ 1 = 10 • , B 1 = 0.9π rad/sample, and T 1 = 0.08π rad/sample, μ = 0.1, th = 0.005, γ = 1.4, and W (ω x , ω u ) = 1 for the passband and W (ω x , ω u ) = 2 for the stopband. We also design the 4-D sparse FIR filters proposed in [21] with the same specifications for α 1 , θ 1 , B 1 and T 1 whereas the hard-thresholding parameter is selected as 0.03 (h th in [21, eq. (5)]). We design the equivalent 4-D nonsparse FIR filters [18] with the same parameters except the hardthresholding parameter, which is zero. We process the "Books," "Flower," "Mirabelle Prune Tree," "Sophie & Vincent 1" and "Gravel Garden" LFs with the proposed filter, the filter proposed in [21], and their equivalent nonsparse filters [18]. We present the average BRISQUE scores, average SSIM indices and the number of nonzero coefficients of the filters in Table I. The central SAIs of the refocused LFs are presented in the supplementary results 2 . According to Table I, it is evident that the proposed 4-D sparse FIR filter provides a better average BRISQUE-score and a similar average-SSIM index compared to the 4-D sparse FIR filter proposed in [21] while providing 13% reduction in computational complexity. The proposed 4-D sparse FIR filter achieves a lower computational complexity compared to that proposed in [21] because the former is an optimal filter whereas the latter is a sub-optimal filter. Furthermore, the proposed sparse filter provides 77% reduction in computational complexity with negligible degradation in the average BRISQUE score and SSIM index compared to an equivalent nonsparse filter [18].

V. CONCLUSION
We demonstrate simultaneous multi-volumetric refocusing of LFs by employing a 4-D sparse FIR filter consisting of multiple hyperfan-shaped passbands. We employ a two-step optimization method to design the optimal 4-D sparse FIR filters in the least-squares sense. Experimental results confirm that the proposed filter provides 63% average reduction in computational complexity with negligible degradation in the fidelity of twovolumetric-region refocused LFs compared to an equivalent 4-D non-sparse FIR filter. Furthermore, in single-volumetric-region refocusing, the proposed filter provides 13% reduction in computational complexity compared to a previously proposed 4-D sparse FIR filters with a negligible degradation in the fidelity.