Computational Photography and Image Manipulation
Instructor: Yağız Aksoy
TA: Sebastian Dille
In this course, we will cover some of the fundamental research topics in computational photography and image manipulation and have a look at the state-of-the-art research going on in the field.
The course will follow a hybrid format that includes live interactive sessions, live-recorded lectures, and pre-recorded lectures. You are expected to attend some of the classes in person over Zoom. Details are in the course schedule below.
It’s probably a good idea for the most of you to brush up on your linear algebra skills as soon as possible to make the best out of this class. 3Blue1Brown has introductory Linear Algebra classes on Youtube with great visual explanations of concepts that we will make use of during the class.
Grading
Programming assignments - 2 x 15% (461) or 3 x 10% (985) = 30%
Paper discussion and video - 20%
Project = 50%
Course Schedule
Weeks 1&2: Computer Vision and Image Processing Fundamentals (Pre-recorded) |
||
Jan 12 |
Jan 15 |
|
Jan 19 |
Jan 22 |
|
Week 3: Cameras and Color (Pre-recorded + Live-recorded) |
||
Jan 26 |
Jan 29 |
|
Extended Cameras Discussion (TBA) |
Color (Live-recorded) |
|
Week 4: HDR, Tonemapping, and Bilateral Filtering (Live-recorded) |
||
Paper assignments |
||
Week 5: Image Blending and Boundary Minimization (Live-recorded) |
||
Project topics announced |
||
February Break |
||
Week 6: Color Propagation (Pre-recorded) |
Quick paper discussions and project updates on Fridays |
|
Week 7: Affinity-Based Matting (Pre-recorded) |
||
Week 8: Color Editing (Pre-recorded) |
||
Week 9: Soft Color Segmentation (Pre-recorded) |
||
Week 10: Spectral Soft Segmentation (Pre-recorded) |
||
Week 11: Practical Applications for Film Editing (Guest lecture, pre-recorded) |
||
Week 12: Selection of Current CPIM Research Problems (Live-recorded) |
||
Week 13: Paper videos and discussions (Live, not recorded) |
||
Programming assignments
There will be 3 programming assignments for 985 (grad) students graded at 10 points each, and 2 programming assignments for 461 (undergrad) at 15 points each. There are no deadlines for these assignments to give you the freedom to do them when it fits your schedule. All assignments has to be submitted by the end of the term, but don't wait until the end to do the assignments! You should be completing them throughout the semester.
Assignment 1: Texture synthesis. CMPT 461 and CMPT 985
Assignment 2: Poisson blending. CMPT 461 and CMPT 985.
Assignment 3: Iterative edge-aware filtering. Only CMPT 985.
Paper reading
Each student will be assigned a recent research paper. You will give a quick overview during one of the Fridays in Weeks 6-10. You will then prepare a video describing the paper. We will talk about how to prepare these videos in detail later in the semester. There will also be a peer-grading excercise in the last week.
For year-by-year listings of SIGGRAPH and SIGGRAPH Asia papers, see Ke-Sen Huang's Home Page .
Gastal and Oliviera,
Spectral Remapping for Image Downscaling,
SIGGRAPH 2017
Heide et al.,
FlexISP: A Flexible Camera Image Processing Framework,
SIGGRAPH Asia 2014
Hasinoff et al.,
Burst photography for high dynamic range and low-light imaging on mobile cameras,
SIGGRAPH Asia 2016
Hui et al.,
Illuminant Spectra-based Source Separation Using Flash Photography
,
CVPR 2018
Shugrina et al.,
Playful Palette: An Interactive Parametric Color Mixer for Artists,
SIGGRAPH 2017
Klose et al.,
Sampling based scene-space video processing,
SIGGRAPH 2015
Ruegg et al.,
DuctTake: Spatiotemporal Video Compositing,
Eurographics 2013
Meka et al.,
Live Intrinsic Video,
SIGGRAPH 2016
Tseng et al.,
Hyperparameter Optimization in Black-box Image Processing using Differentiable Proxies,
SIGGRAPH 2019
Hu et al.,
Exposure: A White-Box Photo Post-Processing Framework,
SIGGRAPH 2018
Zhang et al.,
Synthetic Defocus and Look-Ahead Autofocus for Casual Videography,
SIGGRAPH 2019
Badki et al.,
Computational Zoom: A Framework for Post-Capture Image Composition,
SIGGRAPH 2017
Zhu et al.,
AverageExplorer: Interactive Exploration and Alignment of Visual Data Collections,
SIGGRAPH 2014
Oh et al.,
Personalized Cinemagraphs using Semantic Understanding and Collaborative Learning,
ICCV 2017
Projects
At the second half of the semester, you will do a computational photography project. In these, you will select from a list of project proposals that will be announced later in the semester. These projects will focus on computational photography and image manipulation applications. The project will be done in groups of 2 students.
At the end of the semester, you will prepare a video on your projct and your application results. We will talk about how to prepare these videos in detail later in the semester.
Textbook
There is no required textbook for the course. One useful resource that is also available online for free is the textbook Computer Vision: Algorithms and Applications by Richard Szeliski. There is a great number of resources you can find online, and don't forget that Wikipedia is always your friend.
Announcements, Questions and Discussion
We will use Piazza as our main communication channel. You will receive the sign-up link via CourSys.
Academic Integrity
You are encouraged to talk about and discuss coding assignments and projects with your class-mates. You are allowed to use existing code/library (e.g., optimization library or vector calculus library), in which case, you have to explicitly describe it in your report. Besides the above case, every single line of code must be written by you, and you are not allowed to copy from other sources. Writing the code by exactly or closely following existing code is not technically copy-and-paste, but is also considered to be copy-and-paste. Use your fair judgement. You know what is good and bad. When in doubt, consult the instructor. You are expected to maintain the highest standards of academic integrity and refrain from the forms of misconduct.