Research Overview: Aaron Hertzmann

Broadly speaking, I am interested in all areas of computer graphics and computer vision. My work seeks to address the following high-level questions:

Moreover, I am increasingly interested in applications of Machine Learning — especially Bayesian inference — to these two areas. Computer vision and graphics both rely heavily on analysis and generation of data, and Bayesian learning provides extremely powerful tools for interpreting data. I collaborate actively with my colleagues at UofT. UofT has some of the strongest groups around in the areas of computer vision, machine learning, and graphics/HCI. I also collaborate with a number of labs internationally, including University of Washington's GRAIL lab.

Some specific research areas:

Character Animation by Example
How can we create virtualcharacters from live human performance? Doing so requires building computational models of human motion that incorporate physics of the body as well as an individual's style, explicitly or implicitly. Our work in this area led to the first system for creating animation by example, an area that has since become very popular in the graphics community. We have developed a real-time system for interactive character posing that uses machine learning to help determine which poses are "most likely," and has been licensed to a major game developer. More recently, we have developed some exciting new methods for learning biomechanical human body models. This method can predict how a person will move under new circumstances, based on muscle strengths and stiffnesses estimated from a motion capture sequence.

Collaborators: Brett Allen, Matthew Brand, Seth Cooper, Brian Curless, Keith Grochow, C. Karen Liu, Steven L. Martin, Zoran Popović

Style machines Style IK Learning biomechanics Motion Composition
Learning body shape variation Active learning for mocap

Visual tracking, reconstruction, and rotoscoping
How do we perceive the 3D structure of a video sequence that contains moving people and objects? We have developed several techniques that address aspects of this problem, both addressing fundamental computer vision questions, and also creating a practical application for special effects. In one project, we have developed a method for determining the 3D shape and motion of a non-rigid object from raw video, without any prior knowledge about the deforming object. This is based on our work on probabilistic non-rigid structure-from-motion. We have developed a method for detailed reconstruction of a shaded 3D object from video, for diffusely-reflecting surfaces. We have developed interactive tracking (rotoscoping) techniques that are now in use in the special effects industry.

Collaborators: Aseem Agarwala, Christoph Bregler, Marcus Brubaker, Brian Curless, Alexei A. Efros, David J. Fleet, Pascal Fua, James Hays, Evangelos Kalogerakis, David H. Salesin, Steven M. Seitz, Lorenzo Torresani, Raquel Urtasun, Olga Vesselova, Li Zhang

Automatic non-rigid shape from video Non-rigid SFM Rotoscoping Smooth surfaces from video
Kinematic person tracking Phyics-Based Person Tracking Image Sequence Geolocation

Controllers for physics-based characters
We are developing techniques for creating human and animal motor controllers that move in physically-realistic and expressive ways. Our work is inspired by insights from biology, robotics, and reinforcement learning.

Collaborators: David J. Fleet, Martin de Lasa, Jack M. Wang

Prioritized Optimization Optimizing Walking

Non-photorealistic rendering
How can we write computer software that helps in creating artistic imagery and video? How can we enable computer animation in the styles of human painting and drawing? Answering these questions involves understanding aspects of artistic style and human perception. We have developed technqiues for painterly rendering, painterly animation, and pen-and-ink illustration of 3D surfaces.

Collaborators: Todd Goodwin, Evangelos Kalogerakis, Alex Kolliopoulos, James McCrae, Derek Nowrouzezahrai, Ken Perlin, Patricio Simari, Karan Singh, Ian Vollick, Jack M. Wang, Denis Zorin

Painterly rendering Painterly video Illustrating smooth surfaces Segmentation-Based NPR
Artistic Stroke Thickness Real-Time Curvature

Image processing and texture synthesis by example
How can we creating sophisticated image filters without a lot of difficult programming? We have developed a technique called Image Analogies that learns transformations from user-provided examples. Within a single learning framework, we can learn many different types of transformations, including generating new texture, generating new landscapes, and learning the styles of famous artists like Van Gogh and Manet. We have also developed methods for generating textures on 3D surfaces.

Collaborators: Henning Biermann, Brian Curless, Charles E. Jacobs, Nuria Oliver, Steven M. Seitz, Lexing Ying, Denis Zorin

Image Analogies Surface Texture Synthesis

Reconstructing objects with real-world materials
We have developed methods for 3D shape reconstruction of static objects in the difficult case of complex reflectance (such as shiny objects), cases that foil most shape reconstruction methods, including laser scanners. We have developed a method based on photometric stereo using reference objects that entails very simple setup and calibration, and does not require advance knowledge of shape, illumination, or materials. We have developed a more recent technique that reconstructs BRDFs as well and can be used for relighting and rerendering.

Collaborators: Brian Curless, Dan B. Goldman, Steven M. Seitz, Adrien Treuille

Example-based photometric stereo Example-based multiview stereo Scanning with varying BRDFs

Information visualization

Collaborators: Maneesh Agrawala, Dan Vogel, Ian Vollick

Learning label layout

Single image deblurring
We are investigating Bayesian methods for blind deconvolution of blurred images, such as taken with hand-held cameras.

Collaborators: Rob Fergus, William T. Freeman, Sam T. Roweis, Barun Singh

Single-image deblurring

Machine learning and time-series analysis
We have developed core learning algorithms and applications of learning to various applications, including speech processing, robotic imitation, and time-series analysis. Our techniques typically build on probabilistic inference techniques, such as latent variable models and Gaussian Processes.

Collaborators: Kannan Achan, David J. Fleet, Brendan Frey, Keith Grochow, Rajesh P. N. Rao, Sam T. Roweis, Aaron P. Shon, Jack M. Wang

Gaussian Process Dynamical Models Segmental speech processing Shared Latent GPs Style-Content Gaussian Processes

Aaron Hertzmann