Research Overview: Aaron Hertzmann

Broadly speaking, I am interested in all areas of computer graphics and computer vision. My work seeks to address the following high-level questions:

Moreover, I am interested in applications of Machine Learning to these two areas, as well as applications to Human-Computer Interaction.

Some specific research areas:

Painting and line drawing algorithms
How can we write computer software that helps in creating artistic imagery and video? How can we enable computer animation in the styles of human painting and drawing? Answering these questions involves understanding aspects of artistic style and human perception. We have developed technqiues for painterly rendering, painterly animation, and pen-and-ink illustration of 3D surfaces.

Painterly rendering Painterly video Illustrating smooth surfaces Image Analogies
Paint By Relaxation Curve Analogies Segmentation-Based NPR Artistic Stroke Thickness
Interactive painterly animation Learning Hatching Computing smooth contours PortraitSketch
Controlling Neural Style Line Drawings from 3D Im2Pencil Multidomain Stylization
Video Rigidification Neural Contours

Art, AI, Perception Essays

The Science of Art Can Computers Create Art? Visual Indeterminacy Why Do Line Drawings Work?
Computers Do Not Make Art

Graphic Design and Data-Driven Aesthetics

Learning Label Layout Color Compatibility Learning Single-Page Layout Color personalization
Clip art style similarity Font attributes Recognizing Image Style Infographics style
Interactive layout suggestions Illustration datasets Visual design importance Behance Artistic Media
Stroke-Based Fonts Context-Aware Asset Search LayoutGAN Visual Font Pairing
Design and photo importance Quantify art ambiguity

Learning Human Motion Models
How can we create virtual characters from live human performance? Doing so requires building computational models of human motion that incorporate physics of the body as well as an individual's style, explicitly or implicitly. Our work in this area led to the first system for creating animation by example, an area that has since become very popular in the graphics community. We have developed a real-time system for interactive character posing that uses machine learning to help determine which poses are "most likely," and has been licensed to a major game developer. More recently, we have developed some exciting new methods for learning biomechanical human body models. This method can predict how a person will move under new circumstances, based on muscle strengths and stiffnesses estimated from a motion capture sequence.

Style machines Style IK Learning biomechanics Motion Composition
Shared Latent Gaussian Processes Gaussian Process Dynamical Models Style-Content Gaussian Processes Active learning for mocap

Controllers for simulated locomotion
We are developing techniques for creating human and animal motor controllers that move in physically-realistic and expressive ways. Our work is inspired by insights from biology, robotics, and reinforcement learning.

Prioritized Optimization Optimizing Walking Walking with Uncertainty Feature-Based Controllers
Low-Dimensional Planning Full-Body Spacetime Rotational control

Person tracking and reconstruction
How do we perceive the 3D structure of a video sequence that contains moving people and objects? We have developed several techniques that address aspects of this problem, both addressing fundamental computer vision questions, and also creating a practical application for special effects. In one project, we have developed a method for determining the 3D shape and motion of a non-rigid object from raw video, without any prior knowledge about the deforming object. This is based on our work on probabilistic non-rigid structure-from-motion. We have developed a method for detailed reconstruction of a shaded 3D object from video, for diffusely-reflecting surfaces. We have developed interactive tracking (rotoscoping) techniques that are now in use in the special effects industry.

Non-rigid SFM Automatic non-rigid shape from video Rotoscoping Kinematic person tracking
Physics-Based Person Tracking Hand reconstruction Contact and dynamics estimation

Machine learning for geometry processing

Surface Texture Synthesis Learning body shape variation Real-Time Curvature Learning mesh segmentation
Furniture style Metric Regression Forests Learning segmentation from scraping

Virtual reality video and interfaces

VR Video Editing VR Video Review Depth Conflict Resolution VR Widgets
6-DoF VR video View-Dependent VR Video

Rigid shape reconstruction

Smooth surfaces from video Example-based photometric stereo Example-based multiview stereo Scanning with varying BRDFs
Image-Based Remodeling

Other computer vision and image processing

Single-image deblurring Image Sequence Geolocation Acceptable photographic adjustments Deep image tagging
Portrait Segmentation GAN projection

Other machine learning and data science

Segmental speech processing Latent Factor Travel Model Sparse Gaussian Processes Event sequence visualization
Behance recommendations

Aaron Hertzmann