Computational Photography and Image Manipulation

CMPT 461 / 769, Spring 2023, with full online support

Instructor: Yağız Aksoy

TA: Sebastian Dille

Mo 12:30 – 13:20; Thu 12:30 – 14:20

In this course, we will cover some of the fundamental research topics in computational photography and image manipulation and have a look at the state-of-the-art research going on in the field. The course is offered in-person with an option to participate over Zoom.

Two of the student semester projects were published at SIGGRAPH Posters program in 2022. Check out the SFU CS feature.

This is a complex course consisting of multiple stages. In the first 2 weeks, an overview of image processing and computer vision is provided in flipped-classroom setup. In the following 2 weeks, we cover fundamental computational photography topics in mathematical modeling and also in real-world film production environments through guest lectures. The latter part of the course follows a research-heavy curriculum. 2 hours of each week is dedicated to deep dives to fundamental topics in image manipulation, relating multiple papers on each topic with each other in terms of mathematical modeling and color theory. 1 hour each week is reserved for all-together open-ended discussions on term projects and research papers. Students form project groups and each projects develops their own photography project through discussions with the instructor. Each student also prepares a detailed video presentation of a selected research paper, which is watched and discussed together in class at the end of the semester. In the research-focused lectures, we go over very detailed formulations directly from pdf's of papers. In the discussion hours, the students casually talk about their plans for their project or bring up problems they came across. The students define and create unique applications in project groups. The collaboration is enhanced through weekly check-ins and discussions during lecture hours.

What will you get out of this course?

You will learn

  • fundamental concepts that connect photography and image manipulation to computer vision and image processing.

  • many different aspects and application scenarios of current computational photography research.

  • the math behind important movie post-production techniques such as green-screen keying, HDR tonemapping, and color editing.

  • how to develop a unique computational photography project using cutting edge methods.

  • important image processing techniques that will help you develop better computer vision and computer graphics systems.

  • important mathematical foundations of visual computing such as graph-based formulations, large linear systems, and spectral analysis.

  • effective video making and scientific communication skills that are becoming perhaps as important as technical presentation skills in today's online research world.

It’s probably a good idea for the most of you to brush up on your linear algebra skills as soon as possible to make the best out of this class. 3Blue1Brown has introductory Linear Algebra classes on Youtube with great visual explanations of concepts that we will make use of during the class.

This course builds up on CMPT 361. A refresher on the camera model and image processing fundamentals is advised before taking this course. You can check out this playlist of related topics from CMPT 361 - Intro. Visual Computing here.

COVID-19 Policy:
In order to have a safe learning environment for everyone, we have several guidelines for in-person lectures:
- Please wear a mask, preferably an N-95 mask equivalent or better, if you are attending the live lectures in person.
- If you are feeling sick or you suspect you might have contracted COVID-19, please do not attend the in-person lectures and instead join the Zoom sessions.
- If you have contracted COVID-19, please do not attent the in-person lectures for 2 weeks after your initial diagnosis. Please join the Zoom sessions or follow the lecture recordings.

Student Paper and Project Videos from Past Offerings

Grading

Programming assignments - 2 x 10 (461) or 2 x 5 + 10 (985) = 20%

Paper discussion and video - 30%

Group project = 50%

Tentative Schedule -- expect some changes

Weeks 0&1, Jan 5-12: Entering the Computational Photography research field, background on CV, image processing, photography, video making

Week 2, Jan 16-19: Cameras and Color

Week 3, Jan 23-26: HDR, Tonemapping, and Bilateral Filtering

Paper assignments

Week 4, Jan 30, Feb 2: Image Blending and Boundary Minimization

Project groups formed

Quick paper discussions and project updates on Mondays

Week 5, Feb 6-9: Color Propagation

Week 6, Feb 13-16: Natural Image Matting

Feb 20-23: Reading Break

Week 7, Feb 27-Mar 2: Soft Color Segmentation

Week 8, Mar 6-9: Intrinsic Decomposition

Week 9, Mar 13-16: Monocular Depth Estimation

Week 10, Mar 20-23: Deep Image Editing

Week 11, Mar 27-30: Practical Applications for Film Editing (Guest lecture)

Week 12 Apr 3-6: Current Research through Student Videos

Programming assignments

There will be 3 programming assignments for 985 (grad) students, and 2 programming assignments for 461 (undergrad).

Assignment 1: Texture synthesis. CMPT 461 and CMPT 985

Assignment 2: Poisson blending. CMPT 461 and CMPT 985.

Assignment 3: Iterative edge-aware filtering. Only CMPT 985.

Paper reading

Each student will be assigned a recent research paper. You will give a quick overview during one of the discussion hours on Mondays. You will then prepare a video describing the paper. We will talk about how to prepare these videos in detail later in the semester.

For year-by-year listings of SIGGRAPH and SIGGRAPH Asia papers, see Ke-Sen Huang's Home Page .

Projects

You will develop a computational photography project throughout the semester. These projects will focus on computational photography and image manipulation applications. There will be a set of options for project topics for you to choose from. ou can also develop your own idea. We will design and develop them together through in-class discussions.

At the end of the semester, you will prepare a video on your projct and your application results. We will talk about how to prepare these videos in detail later in the semester.

Textbook

There is no required textbook for the course. One useful resource that is also available online for free is the textbook Computer Vision: Algorithms and Applications by Richard Szeliski. There is a great number of resources you can find online, and don't forget that Wikipedia is always your friend.

Announcements, Questions and Discussion

Make sure to check Coursys for current updates.

Academic Integrity

You are encouraged to talk about and discuss coding assignments and projects with your class-mates. You are allowed to use existing code/library (e.g., optimization library or vector calculus library), in which case, you have to explicitly describe it in your report. Besides the above case, every single line of code must be written by you, and you are not allowed to copy from other sources. Writing the code by exactly or closely following existing code is not technically copy-and-paste, but is also considered to be copy-and-paste. Use your fair judgement. You know what is good and bad. When in doubt, consult the instructor. You are expected to maintain the highest standards of academic integrity and refrain from the forms of misconduct.

Past Offerings

2022 Spring - CMPT 461/985 - Computational Photography and Image Manipulation

2021 Spring - CMPT 461/985 - Computational Photography and Image Manipulation

2019 Fall - CMPT 469/985 - Computational Photography and Image Manipulation (Special Topics)