Computational Photography and Image Manipulation
Instructor: Yağız Aksoy
TA: Sicong Tang
Office hours:
Yağız: Tuesday 2:30pm, following the lecture
Sicong: Monday, 3-4pm, at TASC8004, the small meeting room in GRUVI lab
Prerequisites:
(i) A computer vision, computer graphics or visual computing course
(ii) College-level linear algebra course
In this course, we will cover some of the fundamental research topics in computational photography and image manipulation and have a look at the state-of-the-art research going on in the field. The course will follow a hybrid format that consists of 2-hours of lecture and 1-hour of paper presentations from students every week.
Programming assignments
There will be 4 programming assignments. Students are encouraged to use MATLAB (with the Image Processing Toolkit) for the programming assignments. MATLAB is very easy to use and also makes the grading consistent. You will have 2 weeks to complete each assignment and submit your report and source code.
Assignment 1: Filtering and hybrid images. Due date: Tuesday, October 8, by midnight.
Assignment 2: Iterative edge-aware filtering. Due date: Tuesday, November 5, by midnight.
Assignment 3: Texture synthesis. Due date: Tuesday, November 26, by midnight.
Assignment 4: Poisson blending. Due date: Tuesday, December 10, by midnight.
Tentative schedule
| Sept 3 | Sept 5 |
| Introduction [Slides] | Imaging basics [Slides] |
| Sept 10 | Sept 12 |
| Week assignments [Slides] | Camera basics [Slides] |
| Sept 17 | Sept 19 |
| Frequency domain concepts [Slides] | Fourier transform and sampling [Slides] |
| Sept 24 | Sept 26 |
|
Paper presentations P1, O(9,6,4) Recap and intro to JPEG [Slides] | High dynamic range imaging [Slides] |
| Oct 1 | Oct 3 |
| SFU CS Research Day |
Paper presentations P2, O(10,7,5) Tone mapping and bilateral filtering [Slides] |
| Oct 8 | Oct 10 |
| Paper presentations P3, O(1,8,6) | Color [Slides] |
| Oct 15 | Oct 17 |
| Paper presentations P4, O(2,9,7) | Image blending [Slides] |
| Oct 22 | Oct 24 |
| Paper presentations P5, O(3,10,8) | Boundary minimization techniques [Slides] |
| Oct 29 | Oct 31 |
| Paper presentations P6, O(4,1,9) | Focal stacks and light fields [Slides] |
| Nov 5 | Nov 7 |
| Paper presentations P7, O(5,2,10) | Transformations and panoramas [Slides] |
| Nov 12 | Nov 14 |
| Paper presentations P8, O(6,3,1) | Camera models and optical flow [Slides] |
| Nov 19 | Nov 21 |
| Paper presentations P9, O(7,4,2) | Deconvolution and noise [Slides] |
| Nov 26 | Nov 28 |
| Paper presentations P10, O(8,5,3) | --Cancelled due to public transport strike-- |
Presented papers
P1:
Gastal and Oliviera,
Spectral Remapping for Image Downscaling,
SIGGRAPH 2017
P2:
Isola et al.,
Crisp Boundary Detection Using Pointwise Mutual Information,
ECCV 2014
Raskar et al.,
Coded Exposure Photography: Motion Deblurring using Fluttered Shutter,
SIGGRAPH 2006
P3:
Heide et al.,
FlexISP: A Flexible Camera Image Processing Framework,
SIGGRAPH Asia 2014
Hasinoff et al.,
Burst photography for high dynamic range and low-light imaging on mobile cameras,
SIGGRAPH Asia 2016
Sen et al.,
Robust Patch-Based HDR Reconstruction of Dynamic Scenes,
SIGGRAPH Asia 2012
P4:
Paris et al.,
Local Laplacian Filters: Edge-aware Image Processing with a Laplacian Pyramid,
SIGGRAPH 2011
Chen et al.,
Real-time Edge-Aware Image Processing with the Bilateral Grid,
SIGGRAPH 2007
Aydin et al.,
Temporally Coherent Tone Mapping of HDR Video,
SIGGRAPH Asia 2014
P5:
HaCohen et al.,
Optimizing Color Consistency in Photo Collections,
ACM Transactions on Graphics 2013
Bonneel et al.,
Example-Based Video Color Grading,
ACM Transactions on Graphics 2013
Shugrina et al.,
Playful Palette: An Interactive Parametric Color Mixer for Artists,
SIGGRAPH 2017
P6:
Klose et al.,
Sampling based scene-space video processing,
SIGGRAPH 2015
Ruegg et al.,
DuctTake: Spatiotemporal Video Compositing,
Eurographics 2013
Meka et al.,
Live Intrinsic Video,
SIGGRAPH 2016
P7:
Tseng et al.,
Hyperparameter Optimization in Black-box Image Processing using Differentiable Proxies,
SIGGRAPH 2019
Zhang et al.,
Synthetic Defocus and Look-Ahead Autofocus for Casual Videography,
SIGGRAPH 2019
P8:
Badki et al.,
Computational Zoom: A Framework for Post-Capture Image Composition,
SIGGRAPH 2017
Yucer et al.,
Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction,
ACM Trans. Graph. 2016
P9:
Zhu et al.,
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections,
SIGGRAPH 2014
Arev et al.,
Automatic Editing of Footage from Multiple Social Cameras,
SIGGRAPH 2014
P10:
Bai et al.,
Selectively De-Animating Video,
SIGGRAPH 2012
Oh et al.,
Personalized Cinemagraphs using Semantic Understanding and Collaborative Learning,
ICCV 2017
Grading
Programming assignments - 4 x 15% = 60%
Paper presentation - 30%
Paper discussion - 3 x 3.333333% = 10%
Presentations and paper discussions
Tuesday lectures will consist of paper presentations by students followed by discussions. We will assign a presentation week for everyone on Tuesday the second week. Every week on Thursdays, I will announce the papers for the Tuesday in 2 weeks, so the students will have 12 days to read and understand their assigned paper and prepare a presentation. The presentations will be 8 minutes-long for CMTP 469 students, and 15 minutes-long for CMTP 985 students. The details about these presentations will be discussed at the lecture.
You can propose papers you would like to present on Tuesday the week before your presentation by contacting me. The papers you select should be from SIGGRAPH, SIGGRAPH Asia, Eurographics, Transactions on Graphics, CVPR, ICCV, ECCV or ICCP.
Each presentation will be followed by 4 minutes of discussion. Each discussion will be lead by 3 students. Hence, each student will be assigned one presentation and three discussions that are spread through the semester. The students who will lead the discussions are expected to read and understand the assigned paper, and provide some insights about the strengths/weaknesses of the method, interesting closely-related work, or application scenarios that are beyond what is presented in the paper.
You can switch your assigned weeks for presentations/discussions among each other freely, but make sure to notify me at least 1 week before the assignment.
Late Policy
You have 2 days of penalty-free late submission for the whole semester, use them wisely. If you do submit all the assignments on time, you will receive a 5% bonus to your overall homework grade. After your free 2 days, one-day late submission will receive a 50% penalty. If you submit later than that, you will not receive any points for that homework. Submitting 1 hour or 23 hours after the deadline both count as 1-day late submission.
Textbook
There is no required textbook for the course. One useful resource that is also available online for free is the textbook Computer Vision: Algorithms and Applications by Richard Szeliski. There is a great number of resources you can find online, and don’t forget that Wikipedia is always your friend.
Announcements, Questions and Discussion
Find the course on CourSys: https://coursys.sfu.ca/2019fa-cmpt-985-x2/
Academic Integrity
You are encouraged to talk about and discuss coding assignments and projects with your class-mates. You are allowed to use existing code/library (e.g., optimization library or vector calculus library), in which case, you have to explicitly describe it in your report. Besides the above case, every single line of code must be written by you, and you are not allowed to copy from other sources. Writing the code by exactly or closely following existing code is not technically copy-and-paste, but is also considered to be copy-and-paste. Use your fair judgement. You know what is good and bad. When in doubt, consult the instructor. You are expected to maintain the highest standards of academic integrity and refrain from the forms of misconduct.
Acknowledgements
Computational Photography and Image Manipulation as a class is tought in many institutions with varying flavors. This iteration of the class makes use of material from the classes by James Tompkin, Ioannis Gkioulekas, Marc Pollefeys, and Alyosha Efros. Some of these classes in turn was inspired by / developed from courses / lectures by James Hays, Gordon Wetzstein, Fredo Durand, Mark Levoy, Ramesh Raskar, and many others.