Computational Photography and Image Manipulation
Instructor: Yağız Aksoy
TA: Chris Careaga
Wed 13:30 – 14:20; Fri 12:30 – 14:20
In this course, we will cover some of the fundamental research topics in computational photography and image manipulation and have a look at the state-of-the-art research going on in the field, where students develop custom computational photography applications. The course is offered in-person with an option to participate over Zoom.
Modern computational photography takes its powers from computer vision methods that works at high resolutions in-the-wild and applies them to re-render new photographs using techniques in computer graphics. This course covers computational photography from practical and theoretical perspectives. Students will get an overview of classical and modern formulations in computational photography research and also develop custom practical applications. In the first few weeks, we cover fundamental computational photography concepts, formulations, and applications and also in real-world film production environments through guest lectures. The latter part of the course follows a curriculum focused on recent/current research. 2 hours of each week is dedicated to deep dives to fundamental topics in image manipulation including monocular depth estimation, intrinsic decomposition, soft segmentation and alpha matting, image recoloring, and more. 1 hour each week is reserved for all-together open-ended discussions on term projects and research papers. Students form project groups and each projects develops their own photography project through discussions with the instructor. Each student also prepares a detailed video presentation of a selected research paper, which is watched and discussed together in class at the end of the semester. The collaboration is enhanced through weekly check-ins and discussions during lecture hours.
Attendence to lectures, with in-person and Zoom options, are mandatory for the entire semester.
What will you get out of this course?
You will learn
-
fundamental concepts that connect photography and image manipulation to computer vision and image processing.
-
many different aspects and application scenarios of current computational photography research.
-
the math behind important movie post-production techniques such as green-screen keying, HDR tonemapping, and color editing.
-
how to develop a unique computational photography project using cutting edge methods.
-
important image processing techniques that will help you develop better computer vision and computer graphics systems.
-
important mathematical foundations of visual computing such as graph-based formulations, large linear systems, and spectral analysis.
-
effective video making and scientific communication skills that are becoming perhaps as important as technical presentation skills in today's online research world.
It’s probably a good idea for the most of you to brush up on your linear algebra skills as soon as possible to make the best out of this class. 3Blue1Brown has introductory Linear Algebra classes on Youtube with great visual explanations of concepts that we will make use of during the class.
This course builds up on CMPT 361. A refresher on the camera model and image processing fundamentals is advised before taking this course. You can check out this playlist of related topics from CMPT 361 - Intro. Visual Computing here.
COVID-19 Policy
In order to have a safe learning environment for everyone, we have several guidelines for in-person lectures:
- If you are feeling sick or you suspect you might have contracted COVID-19, please do not attend the in-person lectures and instead join the Zoom sessions.
- If you have contracted COVID-19, please do not attent the in-person lectures for 2 weeks after your initial diagnosis. Please join the Zoom sessions instead.
Grading
Programming assignments - 2 x 10 (461) or 2 x 5 + 10 (769) = 20%
Paper discussion and video - 30%
Group project = 50%
Tentative Schedule -- expect some changes
Week 1: Entering the Computational Photography research field, background on CV, image processing, photography, video making |
|||
Week 2: Cameras and Color |
|||
Week 3: Cameras and Color, HDR, Tonemapping |
|||
Project groups formed |
|||
Week 4: Tonemapping, Bilateral Filtering, Image Blending |
|||
Quick paper discussions and project updates on Wednesdays |
Week 5: Image Blending and Boundary Minimization |
||
Week 6: Natural Image Matting |
|||
Feb 19-25: Reading Break |
|||
Week 7: Soft Color Segmentation |
|||
Week 8: Intrinsic Decomposition |
|||
Week 9: Monocular Depth Estimation |
|||
Week 10: Deep Image Editing |
|||
Week 11: RGB-Space Geometry |
|||
Week 12: Practical Applications for Film Editing (Guest lecture) |
|||
Week 13: Current Research through Student Videos |
|||
Programming assignments
There will be 3 programming assignments for 985 (grad) students, and 2 programming assignments for 461 (undergrad).
Assignment 1: Texture synthesis. CMPT 461 and CMPT 985
Assignment 2: Poisson blending. CMPT 461 and CMPT 985.
Assignment 3: Iterative edge-aware filtering. Only CMPT 985.
Paper reading
Each student will be assigned a recent research paper. You will give a quick overview during one of the discussion hours on Mondays. You will then prepare a video describing the paper. We will talk about how to prepare these videos in detail later in the semester.
For year-by-year listings of SIGGRAPH and SIGGRAPH Asia papers, see Ke-Sen Huang's Home Page .
Projects
You will develop a computational photography project throughout the semester. These projects will focus on computational photography and image manipulation applications. There will be a set of options for project topics for you to choose from. ou can also develop your own idea. We will design and develop them together through in-class discussions.
At the end of the semester, you will prepare a video on your projct and your application results. We will talk about how to prepare these videos in detail later in the semester.
Publications from Past Student Projects
Student-led projects focus on new tools and applications in the computational photography domain and some of them become a great match for SIGGRAPH (Asia) poster programs.SFU CS featured a story on our undergrad authors in 2022.
|
SIGGRAPH Posters, 2022
This work introduces DynaPix, a Krita extension that automatically generates pixelated images and surface normals from an input image.
DynaPix is a tool that aids pixel artists and game developers more efficiently develop 8-bit style games and bring them to life with dynamic lighting through normal maps that can be used in modern game engines such as Unity.
The extension offers artists a degree of flexibility as well as allows for further refinements to generated artwork.
Powered by out of the box solutions, DynaPix is a tool that seamlessly integrates in the artistic workflow.
@INPROCEEDINGS{dynapix,
author={Gerardo Gandeaga and Denys Iliash and Chris Careaga and Ya\u{g}{\i}z Aksoy}, title={Dyna{P}ix: Normal Map Pixelization for Dynamic Lighting}, booktitle={SIGGRAPH Posters}, year={2022}, } |
|
SIGGRAPH Posters, 2022
Art assets for games can be time intensive to produce.
Whether it is a full 3D world, or simpler 2D background, creating good looking assets takes time and skills that are not always readily available.
Time can be saved by using repeating assets, but visible repetition hurts immersion.
Procedural generation techniques can help make repetition less uniform, but do not remove it entirely.
Both approaches leave noticeable levels of repetition in the image, and require significant time and skill investments to produce.
Video game developers in hobby, game jam, or early prototyping situations may not have access to the required time and skill.
We propose a framework to produce layered 2D backgrounds without the need for significant artist time or skill.
In our pipeline, the user provides segmented photographic input, instead of creating traditional art, and receives game-ready assets.
By utilizing photographs as input, we can achieve both a high level of realism for the resulting background texture as well as a shift from manual work away towards computational run-time which frees up developers for other work.
@INPROCEEDINGS{parallaxBG,
author={Brigham Okano and Shao Yu Shen and Sebastian Dille and Ya\u{g}{\i}z Aksoy}, title={Parallax Background Texture Generation}, booktitle={SIGGRAPH Posters}, year={2022}, } |
|
SIGGRAPH Asia Posters, 2023
We propose a simple method for emulating the effect of data moshing, without relying on the corruption of encoded video, and explore its use in different application scenarios.
Like traditional data moshing, we apply motion information to mismatched visual data.
Our approach uses off-the-shelf optical flow estimation to generate motion vectors for each pixel.
Our core algorithm can be implemented in a handful of lines but unlocks multiple video editing effects.
The use of accurate optical flow rather than compression data also creates a more natural transition without block artifacts.
We hope our method provides artists and content creators with more creative freedom over the process of data moshing.
@INPROCEEDINGS{datamosh,
author={Chris Careaga and Mahesh Kumar Krishna Reddy and Ya\u{g}{\i}z Aksoy}, title={Datamoshing with Optical Flow}, booktitle={SIGGRAPH Asia Posters}, year={2023}, } |
Student Paper and Project Videos from Past Offerings
Textbook
There is no required textbook for the course. One useful resource that is also available online for free is the textbook Computer Vision: Algorithms and Applications by Richard Szeliski. There is a great number of resources you can find online, and don't forget that Wikipedia is always your friend.
Announcements, Questions and Discussion
Make sure to check Coursys for current updates.
Academic Integrity
You are encouraged to talk about and discuss coding assignments and projects with your class-mates. You are allowed to use existing code/library (e.g., optimization library or vector calculus library), in which case, you have to explicitly describe it in your report. Besides the above case, every single line of code must be written by you, and you are not allowed to copy from other sources. Writing the code by exactly or closely following existing code is not technically copy-and-paste, but is also considered to be copy-and-paste. Use your fair judgement. You know what is good and bad. When in doubt, consult the instructor. You are expected to maintain the highest standards of academic integrity and refrain from the forms of misconduct.
Past Offerings
2023 Spring - CMPT 461/769 - Computational Photography and Image Manipulation
2022 Spring - CMPT 461/769- Computational Photography and Image Manipulation
2021 Spring - CMPT 461/985 - Computational Photography and Image Manipulation
2019 Fall - CMPT 469/985 - Computational Photography and Image Manipulation (Special Topics)