Computational Photography and Image Manipulation

In this assignment, we will implement the spatial filter proposed by Aydin et al. [2014]. The most relevant parts of this paper for this assignment are Sections 4.1 and 4.2.

At a high-level, you will be implementing an iterative edge-aware filter that follows similar principles to the bilateral filter as we discussed in the lecture. As you should recall, the filtering kernel for the bilateral filter changes from pixel to pixel depending on local texture. We also discussed that it can not be directly implemented as a separable filter like a Gaussian kernel. This paper presents a formulation that uses two separate 1-D edge-aware kernels, one horizontal and one vertical, that can achieve edge-aware smoothing when used iteratively.

Due date: Tuesday, November 5, by midnight

Note: You have some extra time for this assignment due to the absence of our TA for his office hour on October 28. Sicong will be available during his office hour on November 4.

Part 1: Iterative edge-aware filtering (60 pts)

You will first compute the per-pixel edge-aware kernels separately for horizontal and vertical directions. Then you will apply these kernels iteratively to get the final result. For each iteration, you will first filter the image horizontally, and then vertically (i.e. for 5 iterations, you will perform 10 filtering operations).

You will use the 2 images you used in Assignment 1 (the high-frequency and low-frequency ones you selected) to demonstrate your results. In your report, you should show and comment on:
(i) The results after the first horizontal filtering, after the following vertical filtering, and after the second iteration. (input image + 3 partially filtered results).
(ii) The filtering results after 5 iterations using \(\lambda = 0.1\), \(\lambda = 1\), and \(\lambda = 10\).
(ii) The filtering results after 5 iterations using \(W = 5\), \(W = 10\), and \(W = 20\).

See below for the meaning of these parameters.

The spatial part of the filtering explained in the paper involves iterative application of the shift-variant spatial filter \(h^s\) to the image \(J_p^{(k)}\), where \(J_p^{(k)}\) is the result at \(k^{th}\) iteration. The \((k + 1)^{th}\) iteration result is computed as:

\[J_p^{(k+1)} = \sum_{q \in \Omega_p} h_{pq} J_q^{(k)} + \lambda h_{pp} \left( I_p - J_p^{(k)} \right)\]

where \(\lambda\) is a weighting factor that introduces a bias towards the original image. The default value is \(\lambda = 1\) and we will experiment with how this value changes the results. \(I_p\) is the input image at pixel position \(p\), \(J_p^{(k)}\) is the corresponding filtering result after \(k\) iterations with \(J_p^{(0)} = I_p\), \(h_{pq}\) are the filter coefficients, i.e. the spatially-varying kernel, and the set \(\Omega_p\) represents the local neighborhood of \(p\) in one dimension.

Note that \(h_{pq}\) changes between horizontal and vertical filtering operations. The kernels can be computed as follows:

For a pixel \(p = (p_x,p_y)\) and its right neighbor \(p′ = (p_x + 1, p_y)\), the permeability is defined as:

\[\tilde{\pi}_p := \left( 1 + \left\|\frac{I_p - I_{p'}}{\sigma}\right\|^\alpha \right) ^{-1}\]

You can set \(\alpha = 2\) and \(\sigma = 0.5\). The permeability between \(p\) and its right neighbor pixel is close to 0 if the corresponding color difference is high, and it is 1 if the difference is low. For two arbitrary pixels \(p\) and \(q\) on the same row (or column, depending on your filtering direction):

\[\pi_{pq} = \begin{cases} 1, & \text{if}\ p = q \\ \prod_{n=p}^q \tilde\pi_n, & \text{if} |p-q| <= W \\ 0, & \text{otherwise} \end{cases}\]

Here, \(W\) is a user-defined parameter that determines the kernel width. For instance, if \(W = 10\), we have a 21-pixel width kernel (10+10+1). We will set the default value as \(W = 10\) and experiment with how this value changes the results.

To finally compute the coefficients, normalize \(h_{pq}\) so that each per-pixel kernel sums up to \(1\).

In summary, you first need to compute the permeability \(\tilde{\pi}_p\) of each pixel, compute the per-pixel kernels by multiplying these permeabilities to extend the reach of the filter, and then normalize each kernel to sum up to 1. Repeat for both directions.

Part 2: Cross-filtering (40 pts)

The edge-aware filters can use the edges of one image to filter another, typically used in the literature for denoising. For this task, you will compute the kernels using one image and use them to filter another. The image you use to determine the kernels is usually referred to as the guide image.

We will use the iterative filtering approach to denoise no-flash photographs with their pairs taken with flash. This approach was first demonstrated using the bilateral filter by Petschnigg et al.:
Petschnigg et al., Digital Photography with Flash and No-Flash Image Pairs, ACM Trans. Graph. 2004

We will use the flash / no-flash pairs used by Petschnigg et al. [2004] that you can download here. You will use the flash images as the guide to compute the kernels, and filter the no-flash images to get a denoising result. For each image pair in this dataset, you should show:
(i) The filtering result of the no-flash image where the flash image is the guide
(ii) The filtering result where you use the no-flash image to filter itself
(iii) The ‘our_result’ version from the link, which shows the cross-bilateral filtering result
and comment on the differences between these results. You should try to get better results by changing the default parameters we used above, and report the parameters you chose in your report.

Finally, you should show the results for the following and comment:
(i) The cross-filtering result where you use the ‘high-frequency’ image from before as the guide to filter your ‘low-frequency’ image
(ii) The cross-filtering result where you use the ‘low-frequency’ image from before as the guide to filter your ‘high-frequency’ image