Research Overview: Aaron Hertzmann

Broadly speaking, I am interested in all areas of computer graphics and computer vision. My work seeks to address the following high-level questions:

Computer Graphics: What powerful tools can we provide to artists, designers, scientists, and novice users for creating beautiful, expressive, artistic, and/or illustrative imagery and animation?
Computer Vision: How can we visually understand the world, extract meaning from images, and model the human visual system?

Moreover, I am interested in applications of Machine Learning to these two areas, as well as applications to Human-Computer Interaction.

Some specific research areas:

Painting and line drawing algorithms
How can we write computer software that helps in creating artistic imagery and video? How can we enable computer animation in the styles of human painting and drawing? Answering these questions involves understanding aspects of artistic style and human perception. We have developed technqiues for painterly rendering, painterly animation, and pen-and-ink illustration of 3D surfaces.


Painterly rendering	Painterly video	Illustrating smooth surfaces	Image Analogies

Paint By Relaxation	Curve Analogies	Segmentation-Based NPR	Artistic Stroke Thickness

Interactive painterly animation	Learning Hatching	Computing smooth contours	PortraitSketch

Controlling Neural Style	Line Drawings from 3D	Im2Pencil	Multidomain Stylization

Video Rigidification	Neural Contours

Art, AI, Perception Essays


The Science of Art	Can Computers Create Art?	Visual Indeterminacy	Why Do Line Drawings Work?

Computers Do Not Make Art

Graphic Design and Data-Driven Aesthetics


Learning Label Layout	Color Compatibility	Learning Single-Page Layout	Color personalization

Clip art style similarity	Font attributes	Recognizing Image Style	Infographics style

Interactive layout suggestions	Illustration datasets	Visual design importance	Behance Artistic Media

Stroke-Based Fonts	Context-Aware Asset Search	LayoutGAN	Visual Font Pairing

Learning Human Motion Models
How can we create virtual characters from live human performance? Doing so requires building computational models of human motion that incorporate physics of the body as well as an individual's style, explicitly or implicitly. Our work in this area led to the first system for creating animation by example, an area that has since become very popular in the graphics community. We have developed a real-time system for interactive character posing that uses machine learning to help determine which poses are "most likely," and has been licensed to a major game developer. More recently, we have developed some exciting new methods for learning biomechanical human body models. This method can predict how a person will move under new circumstances, based on muscle strengths and stiffnesses estimated from a motion capture sequence.


Style machines	Style IK	Learning biomechanics	Motion Composition

Shared Latent Gaussian Processes	Gaussian Process Dynamical Models	Style-Content Gaussian Processes	Active learning for mocap

Controllers for simulated locomotion
We are developing techniques for creating human and animal motor controllers that move in physically-realistic and expressive ways. Our work is inspired by insights from biology, robotics, and reinforcement learning.


Prioritized Optimization	Optimizing Walking	Walking with Uncertainty	Feature-Based Controllers

Low-Dimensional Planning	Full-Body Spacetime	Rotational control

Non-Rigid/Person tracking and reconstruction
How do we perceive the 3D structure of a video sequence that contains moving people and objects? We have developed several techniques that address aspects of this problem, both addressing fundamental computer vision questions, and also creating a practical application for special effects. In one project, we have developed a method for determining the 3D shape and motion of a non-rigid object from raw video, without any prior knowledge about the deforming object. This is based on our work on probabilistic non-rigid structure-from-motion. We have developed a method for detailed reconstruction of a shaded 3D object from video, for diffusely-reflecting surfaces. We have developed interactive tracking (rotoscoping) techniques that are now in use in the special effects industry.


Non-rigid SFM	Automatic non-rigid shape from video	Rotoscoping	Kinematic person tracking

Physics-Based Person Tracking	Hand reconstruction

Machine learning for geometry processing


Surface Texture Synthesis	Learning body shape variation	Real-Time Curvature	Learning mesh segmentation

Furniture style	Metric Regression Forests	Learning segmentation from scraping