CMPT 361 - Introduction to Visual Computing
In this assignment, you will implement the FAST interest point detector and use it to generate panoramas!
You can do this assignment in any programming language. However, as we cover all these topics with Matlab demonstrations in lectures, it will be easier in Matlab. If you use another language, you need to submit your code with a readme file with detailed explanations on how to run your code (the libraries you used, etc.).
You need to submit your source code and your report through Coursys. Your report will be an html file together with your result images. A template is provided in the report section. The provided template uses these filenames to display your results. Hence, your code must save each result image with the filenames defined below and with a single click.
The MATLAB Image Stitching tutorial we mentioned in the RANSAC lecture will be a helpful resource, although they implement some stuff that is not included in this assignment, even if you are not implementing this assignment in MATLAB.
You have to do this assignment from scratch yourself. If we determine that you have used implementations you found online or elsewhere, even if you modified them, you will get negative points for your assignment due to plagiarism. It is very easy for us to determine if you have used code from others.
1: Take 3 sets of 2 photographs to be stitched together to create a panorama
You need to take photograph pairs to be stitched together, similar to the pair we talked about in our transformations lecture. We provide one set of images for this assignment (namely S1-im1.png and S1-im2.png) and you need to take extra 3 sets of your own. Take each pair of images from different scenes. I would recommend taking many pairs of images and determining which ones to use after some experimentation with your implementation and results. Make sure to resize your images to get the longer dimension of the image (height or width) to be 750.
The images you stitch together have to be different photographs of the same scene. You can not crop an image in different places to stitch them back together.
If you’d like to implement Part 5 for bonus points, at least two of your 3 image sets should contain 4 images to be stitched together.
You can download the first example set from here: S1-im1.png, S1-im2.png
Submit your image pairs named S2-im1.png, S2-im2.png, and so on.
2: FAST feature detector (3 pts.)
Features from accelerated segment test, or FAST, is an efficient interest point detection method proposed by Rosten and Drummond. It works by comparing the brightness of a pixel with a ring surrounding it. You can find more detailed descriptions in the links below:
You need to implement FAST as a function named my_fast_detector. It’s on you to learn about how to define functions in MATLAB.
You can't use pre-implemented detector functions in MATLAB for FAST, it has to be your own implementation.
Hint: In MATLAB, using for loops is quite inefficient but using matrix operations is very efficient. You can do the pixel-wise comparisons by shifting the image for every pixel at once. For example, to check if a pixel is brighter than its neighbor by 3 pixels to the left, shift the image to the right by 3 pixels and compare it against the original image in a single line!
Save the visualization of the detected points in the first images of 2 image sets as S1-fast.png and S2-fast.png. Note that S1-fast should correspond to S1-im1 image provided by us. Your visualization should show the detected points overlaid on the image.
2: Robust FAST using Harris Cornerness metric (1 pts.)
Compute the Harris cornerness measure for each detected FAST feature. Eliminate weak FAST points by defining a threshold for the Harris metric. This threshold must be fixed for every image you use. We will call these points FASTR points.
Hint: Harris cornerness measure is not the same as Harris corners. It is the measure we compute from 2x2 M matrices, not a set of keypoints.
Comment on which points were discarded after this thresholded by comparing your FAST and FASTR visualizations. Save the visualization of the detected points in the first images of your 2 image sets as S1-fastR.png and S2-fastR.png Note down the average computation time of FAST and FASTR features (average of all the images you have) and comment on the difference.
3: Point description and matching (2 pts.)
Use an existing implementation of one of ORB, SURF, or FREAK feature description methods to generate descriptors for your FAST and FASTR points detected in the previous steps. Depending on the function you are using, you might need to put your keypoints in appropriate containers. Don’t forget to use MATLAB Help to get such definitions.
Note which descriptor you use in your report.
Match the features between the first two images in each photo set. Save the visualization of the matched points between the two images of your first 2 image sets, using FAST and FASTR points, as S1-fastMatch.png, S1-fastRMatch.png, S2-fastMatch.png, and S2-fastRMatch.png Comment on the performance differences if any.
4: RANSAC and Panoramas (4 pts.)
To compute the homography between each pair, you will use RANSAC. You can use an existing implementation such as “estgeotform2d” function in MATLAB, but you need to be able to experiment with the RANSAC parameters for the optimum result.
Find the homography between two images in all your image sets and stitch them together.
Experiment with RANSAC parameters and find a setup where you use the minimum number of trials while still getting a satisfactory result for all your image sets. You’ll find 2 different sets of RANSAC parameters, one for FAST and one for FASTR. The RANSAC parameters you decide should be the same for the 4 image sets.
Save the stitched images for all your image sets only using FASTR points, as S1-panorama.png, S2-panorama.png, S3-panorama.png, and S4-panorama.png Comment on the difference between optimal RANSAC parameters for FAST and FASTR.
5: Bonus: Stitch 4 images instead of 2. (2 pts.)
Have 4 images in at least two of your image sets and present panoramas that result in stitching all 4 images. These 4 images can be all horizontally placed (in the panorama it would be [im1 im2 im3 im4] side by side) or placed as a grid (like [im1 im2; im3 im4], the last two images being ‘downstairs neighbors’ of the first two).
6 Report
The report template is here: HTML template
All you need to do is save the images in the current folder as your HTML file and fill in the required information and comments in the HTML file. Don’t edit the template further than needed.
You’ll submit your report as an HTML file in a .zip container together with all the results you have saved. You need to add the necessary comments asked in the question definitions.
There will be a separate placeholder for the code submission on Coursys. Your code submission should run with a single click without any modification and generate the same images as the report. Include any original input images your code needs to run as well as your Matlab scripts in your code submission. If you are using languages other than Matlab include a clear instruction to run your code.
You also have to add the following honor statement to the beginning of your report:
I have not cheated in any way when doing this assignment, I did it on my own. I may have asked questions about the assignment on Coursys, I know that’s totally fine and even encouraged. I also already know that this class is graded on a curve. I realize that if I cheat and by some miracle not get caught, any increase in my grade will in turn shift the curve and result in lower grades for my classmates. Any undeserved extra grade would come at the cost of all others. That’s horrible! I would never do it.