Major venues

S. Mahdi H. Miangoleh*, Sebastian Dille*, Long Mai, Sylvain Paris, and Yağız Aksoy
CVPR, 2021
Neural networks have shown great abilities in estimating depth from a single image. However, the inferred depth maps are well below one-megapixel resolution and often lack fine-grained details, which limits their practicality. Our method builds on our analysis on how the input resolution and the scene structure affects depth estimation performance. We demonstrate that there is a trade-off between a consistent scene structure and the high-frequency details, and merge low- and high-resolution estimations to take advantage of this duality using a simple depth merging network. We present a double estimation method that improves the whole-image depth estimation and a patch selection method that adds local details to the final result. We demonstrate that by merging estimations at different resolutions with changing context, we can generate multi-megapixel depth maps with a high level of detail using a pre-trained model.
author={S. Mahdi H. Miangoleh and Sebastian Dille and Long Mai and Sylvain Paris and Ya\u{g}{\i}z Aksoy},
title={Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging},
journal={Proc. CVPR},

Jingwei Tang, Yağız Aksoy, Cengiz Öztireli, Markus Gross and Tunç Ozan Aydın
CVPR, 2019
The goal of natural image matting is the estimation of opacities of a user-defined foreground object that is essential in creating realistic composite imagery. Natural matting is a challenging process due to the high number of unknowns in the mathematical modeling of the problem, namely the opacities as well as the foreground and background layer colors, while the original image serves as the single observation. In this paper, we propose the estimation of the layer colors through the use of deep neural networks prior to the opacity estimation. The layer color estimation is a better match for the capabilities of neural networks, and the availability of these colors substantially increase the performance of opacity estimation due to the reduced number of unknowns in the compositing equation. A prominent approach to matting in parallel to ours is called sampling-based matting, which involves gathering color samples from known-opacity regions to predict the layer colors. Our approach outperforms not only the previous hand-crafted sampling algorithms, but also current data-driven methods. We hence classify our method as a hybrid sampling- and learning-based approach to matting, and demonstrate the effectiveness of our approach through detailed ablation studies using alternative network architectures.
author={Tang, Jingwei and Aksoy, Ya\u{g}{\i}z and \"Oztireli, Cengiz and Gross, Markus and Ayd{\i}n, Tun\c{c} Ozan},
booktitle={Proc. CVPR},
title={Learning-based Sampling for Natural Image Matting},

Yağız Aksoy, Changil Kim, Petr Kellnhofer, Sylvain Paris, Mohamed Elgharib, Marc Pollefeys and Wojciech Matusik
ECCV, 2018
Illumination is a critical element of photography and is essential for many computer vision tasks. Flash light is unique in the sense that it is a widely available tool for easily manipulating the scene illumination. We present a dataset of thousands of ambient and flash illumination pairs to enable studying flash photography and other applications that can benefi t from having separate illuminations. Different than the typical use of crowdsourcing in generating computer vision datasets, we make use of the crowd to directly take the photographs that make up our dataset. As a result, our dataset covers a wide variety of scenes captured by many casual photographers. We detail the advantages and challenges of our approach to crowdsourcing as well as the computational effort to generate completely separate flash illuminations from the ambient light in an uncontrolled setup. We present a brief examination of illumination decomposition, a challenging and underconstrained problem in flash photography, to demonstrate the use of our dataset in a data-driven approach.
author={Ya\u{g}{\i}z Aksoy and Changil Kim and Petr Kellnhofer and Sylvain Paris and Mohamed Elgharib and Marc Pollefeys and Wojciech Matusik},
booktitle={Proc. ECCV},
title={A Dataset of Flash and Ambient Illumination Pairs from the Crowd},

Yağız Aksoy, Tae-Hyun Oh, Sylvain Paris, Marc Pollefeys and Wojciech Matusik
ACM Transactions on Graphics (Proc. SIGGRAPH), 2018
Accurate representation of soft transitions between image regions is essential for high-quality image editing and compositing. Current techniques for generating such representations depend heavily on interaction by a skilled visual artist, as creating such accurate object selections is a tedious task. In this work, we introduce semantic soft segments, a set of layers that correspond to semantically meaningful regions in an image with accurate soft transitions between different objects. We approach this problem from a spectral segmentation angle and propose a graph structure that embeds texture and color features from the image as well as higher-level semantic information generated by a neural network. The soft segments are generated via eigendecomposition of the carefully constructed Laplacian matrix fully automatically. We demonstrate that otherwise complex image editing tasks can be done with little effort using semantic soft segments.
author={Ya\u{g}{\i}z Aksoy and Tae-Hyun Oh and Sylvain Paris and Marc Pollefeys and Wojciech Matusik},
title={Semantic Soft Segmentation},
journal={ACM Trans. Graph. (Proc. SIGGRAPH)},
pages = {72:1-72:13},
volume = {37},
number = {4}

Yağız Aksoy, Tunç Ozan Aydın and Marc Pollefeys
CVPR, 2017 (spotlight)
We present a novel, purely affinity-based natural image matting algorithm. Our method relies on carefully defined pixel-to-pixel connections that enable effective use of information available in the image and the trimap. We control the information flow from the known-opacity regions into the unknown region, as well as within the unknown region itself, by utilizing multiple definitions of pixel affinities. This way we achieve significant improvements on matte quality near challenging regions of the foreground object. Among other forms of information flow, we introduce color-mixture flow, which builds upon local linear embedding and effectively encapsulates the relation between different pixel opacities. Our resulting novel linear system formulation can be solved in closed-form and is robust against several fundamental challenges in natural matting such as holes and remote intricate structures. While our method is primarily designed as a standalone natural matting tool, we show that it can also be used for regularizing mattes obtained by various sampling-based methods. Our evaluation using the public alpha matting benchmark suggests a significant performance improvement over the state-of-the-art.
author={Aksoy, Ya\u{g}{\i}z and Ayd{\i}n, Tun\c{c} Ozan and Pollefeys, Marc},
booktitle={Proc. CVPR},
title={Designing Effective Inter-Pixel Information Flow for Natural Image Matting},

Yağız Aksoy, Tunç Ozan Aydın, Aljoša Smolić and Marc Pollefeys
ACM Transactions on Graphics, 2017
We present a new method for decomposing an image into a set of soft color segments, which are analogous to color layers with alpha channels that have been commonly utilized in modern image manipulation software. We show that the resulting decomposition serves as an effective intermediate image representation, which can be utilized for performing various, seemingly unrelated image manipulation tasks. We identify a set of requirements that soft color segmentation methods have to fulfill, and present an in-depth theoretical analysis of prior work. We propose an energy formulation for producing compact layers of homogeneous colors and a color refinement procedure, as well as a method for automatically estimating a statistical color model from an image. This results in a novel framework for automatic and high-quality soft color segmentation, which is efficient, parallelizable, and scalable. We show that our technique is superior in quality compared to previous methods through quantitative analysis as well as visually through an extensive set of examples. We demonstrate that our soft color segments can easily be exported to familiar image manipulation software packages and used to produce compelling results for numerous image manipulation applications without forcing the user to learn new tools and workflows.
author={Ya\u{g}{\i}z Aksoy and Tun\c{c} Ozan Ayd{\i}n and Aljo\v{s}a Smoli\'{c} and Marc Pollefeys},
title={Unmixing-Based Soft Color Segmentation for Image Manipulation},
journal={ACM Trans. Graph.},
pages = {19:1-19:19},
volume = {36},
number = {2}

Yağız Aksoy, Tunç Ozan Aydın, Marc Pollefeys and Aljoša Smolić
ACM Transactions on Graphics, 2016
Due to the widespread use of compositing in contemporary feature films, green-screen keying has become an essential part of post-production workflows. To comply with the ever-increasing quality requirements of the industry, specialized compositing artists spend countless hours using multiple commercial software tools, while eventually having to resort to manual painting because of the many shortcomings of these tools. Due to the sheer amount of manual labor involved in the process, new green-screen keying approaches that produce better keying results with less user interaction are welcome additions to the compositing artist's arsenal. We found that --- contrary to the common belief in the research community --- production-quality green-screen keying is still an unresolved problem with its unique challenges. In this paper, we propose a novel green-screen keying method utilizing a new energy minimization-based color unmixing algorithm. We present comprehensive comparisons with commercial software packages and relevant methods in literature, which show that the quality of our results is superior to any other currently available green-screen keying solution. Importantly, using the proposed method, these high-quality results can be generated using only one-tenth of the manual editing time that a professional compositing artist requires to process the same content having all previous state-of-the-art tools at his disposal.
author={Ya\u{g}{\i}z Aksoy and Tun\c{c} Ozan Ayd{\i}n and Marc Pollefeys and Aljo\v{s}a Smoli\'{c}},
title={Interactive High-Quality Green-Screen Keying via Color Unmixing},
journal={ACM Trans. Graph.},
volume = {35},
number = {5},
pages = {152:1--152:12},

Minors, neighbors

Theo Thonat, Yağız Aksoy, Miika Aittala, Sylvain Paris, Frédo Durand, and George Drettakis
Computer Graphics Forum (Proc. EGSR), 2021
Image-Based Rendering allows users to easily capture a scene using a single camera and then navigate freely with realistic results. However, the resulting renderings are completely static, and dynamic effects – such as fire, waterfalls or small waves – cannot be reproduced. We tackle the challenging problem of enabling free-viewpoint navigation including such stationary dynamic effects, but still maintaining the simplicity of casual capture. Using a single camera – instead of previous complex synchronized multi-camera setups – means that we have unsynchronized videos of the dynamic effect from multiple views, making it hard to blend them when synthesizing novel views. We present a solution that allows smooth free-viewpoint video-based rendering (VBR) of such scenes using temporal Laplacian pyramid decomposition video, enabling spatio-temporal blending. For effects such as fire and waterfalls, that are semi-transparent and occupy 3D space, we first estimate their spatial volume. This allows us to create per-video geometries and alpha-matte videos that we can blend using our frequency-dependent method. We also extend Laplacian blending to the temporal dimension to remove additional temporal seams. We show results on scenes containing fire, waterfalls or rippling waves at the seaside, bringing these scenes to life.
author={Thonat, Theo and Aksoy, Ya\u{g}{\i}z and Aittala, Miika and Paris, Sylvain and Durand, Fr\'edo and Drettakis, George},
title={Video-Based Rendering of Dynamic Stationary Environments from Unsynchronized Inputs},
journal={Computer Graphics Forum (Proc. EGSR)},
volume = {40},
number = {4}

Alexandre Kaspar, Geneviève Patterson, Changil Kim, Yağız Aksoy, Wojciech Matusik and Mohamed Elgharib
ACM CHI Conference on Human Factors in Computing Systems, 2018
In this work, we propose two ensemble methods leveraging a crowd workforce to improve video annotation, with a focus on video object segmentation. Their shared principle is that while individual candidate results may likely be insufficient, they often complement each other so that they can be combined into something better than any of the individual results - the very spirit of collaborative working. For one, we extend a standard polygon-drawing interface to allow workers to annotate negative space, and combine the work of multiple workers instead of relying on a single best one as commonly done in crowdsourced image segmentation. For the other, we present a method to combine multiple automatic propagation algorithms with the help of the crowd. Such combination requires an understanding of where the algorithms fail, which we gather using a novel coarse scribble video annotation task. We evaluate our ensemble methods, discuss our design choices for them, and make our web-based crowdsourcing tools and results publicly available.
author={Alexandre Kaspar and Genevi\`eve Patterson and Changil Kim and Ya\u{g}{\i}z Aksoy and Wojciech Matusik and Mohamed Elgharib},
title={Crowd-Guided Ensembles: How Can We Choreograph Crowd Workers for Video Segmentation?},
booktitle={Proc. ACM CHI},


Yağız Aksoy
PhD Thesis, ETH Zurich, 2019
author={Ya\u{g}{\i}z Aksoy},
title={Soft Segmentation of Images},
school={ETH Zurich},

Yağız Aksoy
Master's Thesis, Middle East Technical University, 2013
With the increase in the number and computational power of commercial mobile devices like smart phones and tablet computers, augmented reality applications are gaining more and more volume. In order to augment virtual objects effectively in real scenes, pose of the camera should be estimated with high precision and speed. Today, most of the mobile devices feature cameras and inertial measurement units which carry information on change in position and attitude of the camera. In this thesis, utilization of inertial sensors on mobile devices in aiding visual pose estimation is studied. Error characteristics of the inertial sensors on the utilized mobile device are analyzed. Gyroscope readings are utilized for aiding 2D feature tracking while accelerometer readings are used to help create a sparse 3D map of features later to be used for visual pose estimation. Metric velocity estimation is formulated using inertial readings and observations of a single 2D feature. Detailed formulations of uncertainties on all the estimated variables are provided. Finally, a novel, lightweight filter, which is capable of estimating the pose of the camera together with the metric scale, is proposed. The proposed filter runs without any heuristics needed for covariance propagation, which enables it to be used in different devices with different sensor characteristics without any modifications. Sensors with poor noise characteristics are successfully utilized to aid the visual pose estimation.
author = {Aksoy, Ya\u{g}{\i}z},
title = {Efficient Inertially Aided Visual Odometry towards Mobile Augmented Reality},
school = {Middle East Technical University},
year = {2013},
month = {August}}

Other conferences and workshops

Mattia Ryffel, Fabio Zünd, Yağız Aksoy, Alessia Marra, Maurizio Nitti, Tunç Ozan Aydın and Bob Sumner
International Conference on Game and Entertainment Technologies, 2017
We present a mobile augmented reality application that allows its users to modify colors of paintings via simple touch interactions. Our method is intended for museums and art exhibitions and aims to provide an entertaining way for interacting with paintings in a non-intrusive manner. Plausible color edits are achieved by utilizing a set of layers with corresponding alpha channels, which needs to be generated for each individual painting in a pre-processing step. Manually performing such a layer decomposition is a tedious process and makes the entire system infeasible for most practical use cases. In this work, we propose the use of a fully automatic soft color segmentation algorithm for content generation for such an augmented reality application. This way, we significantly reduce the amount of manual labor needed for deploying our system and thus make our system feasible for real-world use.
author={Mattia Ryffel and Fabio Z\"und and Ya\u{g}{\i}z Aksoy and Alessia Marra and Maurizio Nitti and Tun\c{c} Ozan Ayd{\i}n and Bob Sumner},
title={AR Museum: A Mobile Augmented Reality Application for Interactive Painting Recoloring},
booktitle={International Conference on Game and Entertainment Technologies},

Yağız Aksoy and A. Aydın Alatan
Most of the mobile applications require efficient and precise computation of the device pose, and almost every mobile device has inertial sensors already equipped together with a camera. This fact makes sensor fusion quite attractive for increasing efficiency during pose tracking. However, the state-of-the-art fusion algorithms have a major shortcoming: lack of well-defined uncertainty introduced to the system during the prediction stage of the fusion filters. Such a drawback results in determining covariances heuristically, and hence, requirement for data-dependent tuning to achieve high performance or even convergence of these filters. In this paper, we propose an inertially-aided visual odometry system that requires neither heuristics nor parameter tuning; computation of the required uncertainties on all the estimated variables are obtained after minimum number of assumptions. Moreover, the proposed system simultaneously estimates the metric scale of the pose computed from a monocular image stream. The experimental results indicate that the proposed scale estimation outperforms the state-of-the-art methods, whereas the pose estimation step yields quite acceptable results in real-time on resource constrained systems.
author={Aksoy, Ya\u{g}{\i}z and Alatan, A. Ayd{\i}n},
title={Uncertainty Modeling for Efficient Visual Odometry via Inertial Sensors on Mobile Devices},
booktitle={IEEE International Conference on Image Processing (ICIP)},

Florian Angehrn, Oliver Wang, Yağız Aksoy, Markus Gross and Aljosa Smolic
Free viewpoint video enables interactive viewpoint selection in real world scenes, which is attractive for many applications such as sports visualization. Multi-camera registration is one of the difficult tasks in such systems. We introduce the concept of a static high resolution master camera for improved long-term multiview alignment. All broadcast cameras are aligned to a common reference. Our approach builds on frame-to-frame alignment, extended into a recursive long-term estimation process, which is shown to be accurate, robust and stable over long sequences.
author={Florian Angehrn and Oliver Wang and Ya\u{g}{\i}z Aksoy and Markus Gross and Aljo\v{s}a Smoli\'{c}},
title={MasterCam FVV: Robust Registration of Multiview Sports Video to a Static High-Resolution Master Camera for Free Viewpoint Video},
booktitle={IEEE International Conference on Image Processing (ICIP)},

Yağız Aksoy and A. Aydın Alatan
ECCV Workshops, 2012
Shadows are illuminated as a result of Rayleigh scattering phenomenon, which happens to be more effective for small wavelengths of light. We propose utilization of false color images for shadow detection, since the transformation eliminates high frequency blue component and introduces low frequency near-infrared channel. Effectiveness of the approach is tested by using several shadow-variant texture and color-related cues proposed in the literature. Performances of these cues in regular and false color images are compared and analyzed within a supervised system by using a support vector machine classifier.
author={Aksoy, Yagiz and Alatan, A. Aydin},
title={Utilization of False Color Images in Shadow Detection},
booktitle={European Conference on Computer Vision (ECCV) Workshops},

Yağız Aksoy, Ozan Şener, A. Aydın Alatan and Kemal Uğur
We propose a complete still image based 2D-3D mobile conversion system for touch screen use. The system consists of interactive segmentation followed by 3D rendering. The interactive segmentation is conducted dynamically by color Gaussian mixture model updates and dynamic-iterative graph-cut. A coloring gesture is used to guide the way and entertain the user during the process. Output of the image segmentation is then fed to the 3D rendering stage of the system. For rendering stage, two novel improvements are proposed to handle holes resulting from depth image based rendering process. These improvements are also expected to enhance the 3D perception. These two methods are subjectively tested and their results are presented.
author={Aksoy, Yagiz and Sener, Ozan and Alatan, A. Aydin and Ugur, Kemal},
title={Interactive 2D-3D Image Conversion for Mobile Devices},
booktitle={IEEE International Conference on Image Processing (ICIP)},