Research Overview: Aaron Hertzmann

Broadly speaking, I am interested in all areas of computer graphics and computer vision. My work seeks to address the following high-level questions:

Moreover, I am increasingly interested in applications of Machine Learning — especially Bayesian inference — to these two areas. Computer vision and graphics both rely heavily on analysis and generation of data, and Bayesian learning provides extremely powerful tools for interpreting data. I collaborate actively with my colleagues at UofT. UofT has some of the strongest groups around in the areas of computer vision, machine learning, and graphics/HCI. I also collaborate with a number of labs internationally, including University of Washington's GRAIL lab.

Some specific research areas:

Painting and line drawing algorithms
How can we write computer software that helps in creating artistic imagery and video? How can we enable computer animation in the styles of human painting and drawing? Answering these questions involves understanding aspects of artistic style and human perception. We have developed technqiues for painterly rendering, painterly animation, and pen-and-ink illustration of 3D surfaces.

Collaborators: Pierre Bénard, Matthias Bethge, Simon Breslav, Brian Curless, Alexander S. Ecker, Leon Gatys, Todd Goodwin, Charles E. Jacobs, Evangelos Kalogerakis, Michael Kass, Alex Kolliopoulos, Wilmot Li, Derek Nowrouzezahrai, Peter O'Donovan, Nuria Oliver, Ken Perlin, David H. Salesin, Steven M. Seitz, Eli Shechtman, Ian Vollick, Jack M. Wang, Holger Winnemöller, Jun Xie, Denis Zorin

Painterly rendering Painterly video Illustrating smooth surfaces Image Analogies
Paint By Relaxation Curve Analogies Segmentation-Based NPR Artistic Stroke Thickness
Interactive painterly animation Learning Hatching Computing smooth contours PortraitSketch
Controlling Neural Style

Art, Design, and Aesthetics

Collaborators: Aseem Agarwala, Maneesh Agrawala, Sami Alsheikh, Zoya Bylinskii, Trevor Darrell, Mira Dontcheva, Frédo Durand, Elena Garces, Diego Gutierrez, Helen Han, Sergey Karayev, Nam Wook Kim, Jānis Lībeks, Zhicheng Liu, Spandan Madan, Peter O'Donovan, Hanspeter Pfister, Bryan C. Russell, Babak Saleh, Matthew Trentacoste, Dan Vogel, Ian Vollick, Holger Winnemöller

The Science of Art Learning Label Layout Color Compatibility Learning Single-Page Layout
Color personalization Clip art style similarity Font attributes Recognizing Image Style
Infographics style Interactive layout suggestions Illustration datasets Visual design importance

Learning Human Motion Models
How can we create virtual characters from live human performance? Doing so requires building computational models of human motion that incorporate physics of the body as well as an individual's style, explicitly or implicitly. Our work in this area led to the first system for creating animation by example, an area that has since become very popular in the graphics community. We have developed a real-time system for interactive character posing that uses machine learning to help determine which poses are "most likely," and has been licensed to a major game developer. More recently, we have developed some exciting new methods for learning biomechanical human body models. This method can predict how a person will move under new circumstances, based on muscle strengths and stiffnesses estimated from a motion capture sequence.

Collaborators: Matthew Brand, Seth Cooper, David J. Fleet, Keith Grochow, C. Karen Liu, Steven L. Martin, Zoran Popović, Rajesh P. N. Rao, Aaron P. Shon, Jack M. Wang

Style machines Style IK Learning biomechanics Motion Composition
Shared Latent Gaussian Processes Gaussian Process Dynamical Models Style-Content Gaussian Processes Active learning for mocap

Video tracking, reconstruction, and rotoscoping
How do we perceive the 3D structure of a video sequence that contains moving people and objects? We have developed several techniques that address aspects of this problem, both addressing fundamental computer vision questions, and also creating a practical application for special effects. In one project, we have developed a method for determining the 3D shape and motion of a non-rigid object from raw video, without any prior knowledge about the deforming object. This is based on our work on probabilistic non-rigid structure-from-motion. We have developed a method for detailed reconstruction of a shaded 3D object from video, for diffusely-reflecting surfaces. We have developed interactive tracking (rotoscoping) techniques that are now in use in the special effects industry.

Collaborators: Aseem Agarwala, Christoph Bregler, Marcus Brubaker, Michael F. Cohen, Alex Colburn, Brian Curless, Andrew Fitzgibbon, David J. Fleet, Pascal Fua, Shahram Izadi, Cem Keskin, Gerard Pons-Moll, Varun Ramakrishna, David H. Salesin, Steven M. Seitz, Jamie Shotton, Richard Stebbing, Jonathan Taylor, Lorenzo Torresani, Raquel Urtasun, Li Zhang

Non-rigid SFM Automatic non-rigid shape from video Rotoscoping Smooth surfaces from video
Kinematic person tracking Physics-Based Person Tracking Image-Based Remodeling Metric Regression Forests
Hand reconstruction

Controllers for simulated locomotion
We are developing techniques for creating human and animal motor controllers that move in physically-realistic and expressive ways. Our work is inspired by insights from biology, robotics, and reinforcement learning.

Collaborators: Mazen Al Borno, Eugene Fiume, David J. Fleet, Martin de Lasa, Igor Mordatch, Jack M. Wang

Prioritized Optimization Optimizing Walking Walking with Uncertainty Feature-Based Controllers
Low-Dimensional Planning Full-Body Spacetime Rotational control

Machine learning for geometry processing

Collaborators: Brett Allen, Henning Biermann, Brian Curless, Thomas Funkhouser, Leo Guibas, Evangelos Kalogerakis, Vladimir G. Kim, Wilmot Li, Tianqiang Liu, James McCrae, Derek Nowrouzezahrai, Zoran Popović, Patricio Simari, Karan Singh, Hao Su, Li Yi, Lexing Ying, Ersin Yumer, Denis Zorin

Surface Texture Synthesis Learning body shape variation Real-Time Curvature Learning mesh segmentation
Furniture style Learning segmentation from scraping

Image Processing and Recognition

Collaborators: Vladimir Bychkovsky, Frédo Durand, Alexei A. Efros, Ali Farhadi, Rob Fergus, William T. Freeman, James Hays, Matthew D. Hoffman, Hamid Izadinia, Ronnachai Jaroensri, Jiaya Jia, Evangelos Kalogerakis, Sylvain Paris, Brian Price, Sam T. Roweis, Bryan C. Russell, Ian Sachs, Eli Shechtman, Xiaoyong Shen, Barun Singh, Olga Vesselova

Single-image deblurring Image Sequence Geolocation Acceptable photographic adjustments Deep image tagging
Portrait Segmentation

Reconstructing objects with real-world materials
We have developed methods for 3D shape reconstruction of static objects in the difficult case of complex reflectance (such as shiny objects), cases that foil most shape reconstruction methods, including laser scanners. We have developed a method based on photometric stereo using reference objects that entails very simple setup and calibration, and does not require advance knowledge of shape, illumination, or materials. We have developed a more recent technique that reconstructs BRDFs as well and can be used for relighting and rerendering.

Collaborators: Brian Curless, Dan B Goldman, Steven M. Seitz, Adrien Treuille

Example-based photometric stereo Example-based multiview stereo Scanning with varying BRDFs

Other machine learning/data science

Collaborators: Kannan Achan, Marcus Brubaker, Yanshuai Cao, Mira Dontcheva, David J. Fleet, Brendan Frey, Michael Guerzhoy, Matthew D. Hoffman, Zhicheng Liu, Sam T. Roweis, Maja Rudolph, Alan Wilson, Jian Zhao

Segmental speech processing Latent Factor Travel Model Sparse Gaussian Processes Event sequence visualization
Behance recommendations

Virtual Reality user interfaces

Collaborators: Stephen DiVerdi, Feng Liu, Cuong Nguyen

VR Video Editing CollaVR

Aaron Hertzmann