Computational Photography and Image Manipulation

CMPT 461 / 985 / 769, Spring 2022, ONLINE COURSE

Instructor: Yağız Aksoy

TA: Sebastian Dille

Mo 4:30 PM – 6:20 PM; We 4:30 PM – 5:20 PM; over Zoom

In this course, we will cover some of the fundamental research topics in computational photography and image manipulation and have a look at the state-of-the-art research going on in the field. The course is offered online with live lectures over Zoom.

This is a complex course consisting of multiple stages. In the first 2 weeks, an overview of image processing and computer vision is provided in flipped-classroom setup. In the following 2 weeks, we cover fundamental computational photography topics in mathematical modeling and also in real-world film production environments through guest lectures. The latter part of the course follows a research-heavy curriculum. 2 hours of each week is dedicated to deep dives to fundamental topics in image manipulation, relating multiple papers on each topic with each other in terms of mathematical modeling and color theory. 1 hour each week is reserved for all-together open-ended discussions on term projects and research papers. Students form project groups and each projects develops their own photography project through discussions with the instructor. Each student also prepares a detailed video presentation of a selected research paper, which is watched and discussed together in class at the end of the semester. Flipped classroom setup in overview weeks allows for covering any open questions of students on fundamental topics, and also discussing the research field in general, introducing coding tools for photography etc. This is done in an open environment where every student participates by sharing links, sharing screen to show examples etc. In the research-focused lectures, we go over very detailed formulations directly from pdf's of papers. A tablet + Zoom setup allows a more adaptive lecture coverage guided by student questions or in-class discussions. In the discussion hours, the students casually talk about their plans for their project or bring up problems they came across. The Zoom setup with easy screen-sharing and link sharing makes this discussion very fruitful. The students define and create unique applications in project groups. The collaboration is enhanced through weekly check-ins and discussions during lecture hours.

What will you get out of this course?

You will learn

  • fundamental concepts that connect photography and image manipulation to computer vision and image processing.

  • many different aspects and application scenarios of current computational photography research.

  • the math behind important movie post-production techniques such as green-screen keying, HDR tonemapping, and color editing.

  • how to develop a unique computational photography project using cutting edge methods.

  • important image processing techniques that will help you develop better computer vision and computer graphics systems.

  • important mathematical foundations of visual computing such as graph-based formulations, large linear systems, and spectral analysis.

  • effective video making and scientific communication skills that are becoming perhaps as important as technical presentation skills in today's online research world.

It’s probably a good idea for the most of you to brush up on your linear algebra skills as soon as possible to make the best out of this class. 3Blue1Brown has introductory Linear Algebra classes on Youtube with great visual explanations of concepts that we will make use of during the class.

This course builds up on CMPT 361. A refresher on the camera model and image processing fundamentals is advised before taking this course. You can check out this playlist of related topics from CMPT 361 - Intro. Visual Computing here.

Student Paper and Project Videos from Past Offerings

Grading

Programming assignments - 2 x 10 (461) or 2 x 5 + 10 (985) = 20%

Paper discussion and video - 30%

Group project = 50%

Tentative Schedule -- expect some changes

Weeks 1&2: Entering the Computational Photography research field, background on CV, image processing, photography, video making

Week 3: Cameras and Color

Week 4: HDR, Tonemapping, and Bilateral Filtering

Paper assignments

Week 5: Image Blending and Boundary Minimization

Project groups formed

February Break

Week 6: Color Propagation

Quick paper discussions and project updates on Wednesdays

Week 7: Affinity-Based Matting

Week 8: Color Editing

Week 9: Soft Color Segmentation

Week 10: Spectral Soft Segmentation

Week 11: Practical Applications for Film Editing (Guest lecture)

Week 12: Paper videos and discussions

Week 13: Project videos and discussions

Programming assignments

There will be 3 programming assignments for 985 (grad) students, and 2 programming assignments for 461 (undergrad).

Assignment 1: Texture synthesis. CMPT 461 and CMPT 985

Assignment 2: Poisson blending. CMPT 461 and CMPT 985.

Assignment 3: Iterative edge-aware filtering. Only CMPT 985.

Paper reading

Each student will be assigned a recent research paper. You will give a quick overview during one of the Fridays in Weeks 6-10. You will then prepare a video describing the paper. We will talk about how to prepare these videos in detail later in the semester.

For year-by-year listings of SIGGRAPH and SIGGRAPH Asia papers, see Ke-Sen Huang's Home Page .

Projects

At the second half of the semester, you will develop a computational photography project. These projects will focus on computational photography and image manipulation applications. We will design and develop them together through in-class discussions.

At the end of the semester, you will prepare a video on your projct and your application results. We will talk about how to prepare these videos in detail later in the semester.

Textbook

There is no required textbook for the course. One useful resource that is also available online for free is the textbook Computer Vision: Algorithms and Applications by Richard Szeliski. There is a great number of resources you can find online, and don't forget that Wikipedia is always your friend.

Announcements, Questions and Discussion

Make sure to check Coursys for current updates.

Academic Integrity

You are encouraged to talk about and discuss coding assignments and projects with your class-mates. You are allowed to use existing code/library (e.g., optimization library or vector calculus library), in which case, you have to explicitly describe it in your report. Besides the above case, every single line of code must be written by you, and you are not allowed to copy from other sources. Writing the code by exactly or closely following existing code is not technically copy-and-paste, but is also considered to be copy-and-paste. Use your fair judgement. You know what is good and bad. When in doubt, consult the instructor. You are expected to maintain the highest standards of academic integrity and refrain from the forms of misconduct.

Past Offerings

2021 Spring - CMPT 461/985 - Computational Photography and Image Manipulation

2019 Fall - CMPT 469/985 - Computational Photography and Image Manipulation (Special Topics)