Broadly speaking, I am interested in all areas of computer graphics and computer vision. My work seeks to address the following high-level questions:
Computer Graphics: What powerful tools can we provide to artists, designers, scientists, and novice users for creating beautiful, expressive, artistic, and/or illustrative imagery and animation?
Computer Vision: How can we visually understand the world, extract meaning from images, and model the human visual system?
Moreover, I am interested in applications of Machine Learning to these two areas, as well as applications to Human-Computer Interaction.
Some specific research areas:
Painting and line drawing algorithms
How can we write computer software that helps in creating artistic imagery and video? How can we enable computer animation in the styles of human painting and drawing? Answering these questions involves understanding aspects of artistic style and human perception. We have developed technqiues for painterly rendering, painterly animation, and pen-and-ink illustration of 3D surfaces.
Collaborators: Pierre Bénard, Matthias Bethge, Adrien Bousseau, Simon Breslav, Brian Curless, Johanna Delanoy, Alexander S. Ecker, Chen Fang, Leon Gatys, Todd Goodwin, Charles E. Jacobs, Hailin Jin, Evangelos Kalogerakis, Michael Kass, Alex Kolliopoulos, Yijun Li, Wilmot Li, Derek Nowrouzezahrai, Peter O'Donovan, Nuria Oliver, Ken Perlin, David H. Salesin, Steven M. Seitz, Eli Shechtman, Ian Vollick, Jack M. Wang, Michael J. Wilber, Holger Winnemöller, Jun Xie, Zheng Xu, Ming-Hsuan Yang, Denis Zorin
|Painterly rendering||Painterly video||Illustrating smooth surfaces||Image Analogies|
|Paint By Relaxation||Curve Analogies||Segmentation-Based NPR||Artistic Stroke Thickness|
|Interactive painterly animation||Learning Hatching||Computing smooth contours||PortraitSketch|
|Controlling Neural Style||Line Drawings from 3D||Im2Pencil||Multidomain Stylization|
Art, Design, and Aesthetics
Collaborators: Aseem Agarwala, Maneesh Agrawala, Sami Alsheikh, Kavita Bala, Elena Balashova, Serge Belongie, Amit H. Bermano, Zoya Bylinskii, John Collomosse, Trevor Darrell, Stephen DiVerdi, Mira Dontcheva, Frédo Durand, Chen Fang, Thomas Funkhouser, Elena Garces, Diego Gutierrez, Helen Han, Hailin Jin, Sergey Karayev, Vladimir G. Kim, Nam Wook Kim, Balazs Kovacs, Jānis Lībeks, Jianan Li, Zhicheng Liu, Spandan Madan, Peter O'Donovan, Hanspeter Pfister, Bryan C. Russell, Babak Saleh, Matthew Trentacoste, Dan Vogel, Ian Vollick, Michael J. Wilber, Holger Winnemöller, Tingfa Xu, Jimei Yang, Jianming Zhang
|The Science of Art||Learning Label Layout||Color Compatibility||Learning Single-Page Layout|
|Color personalization||Clip art style similarity||Font attributes||Recognizing Image Style|
|Infographics style||Interactive layout suggestions||Illustration datasets||Visual design importance|
|Behance Artistic Media||Can Computers Create Art?||Stroke-Based Fonts||Context-Aware Asset Search|
|LayoutGAN||Neural Network Aesthetics|
Virtual reality video and interfaces
Collaborators: Zhili Chen, Stephen DiVerdi, Diego Gutierrez, Incheol Kim, Feng Liu, Belen Masia, Cuong Nguyen, Ana Serrano, Ying-Chao Tung, Jacob O. Wobbrock
|VR Video Editing||VR Video Review||Depth Conflict Resolution||VR Widgets|
|6-DoF VR video|
Learning Human Motion Models
How can we create virtual characters from live human performance? Doing so requires building computational models of human motion that incorporate physics of the body as well as an individual's style, explicitly or implicitly. Our work in this area led to the first system for creating animation by example, an area that has since become very popular in the graphics community. We have developed a real-time system for interactive character posing that uses machine learning to help determine which poses are "most likely," and has been licensed to a major game developer. More recently, we have developed some exciting new methods for learning biomechanical human body models. This method can predict how a person will move under new circumstances, based on muscle strengths and stiffnesses estimated from a motion capture sequence.
Collaborators: Matthew Brand, Seth Cooper, David J. Fleet, Keith Grochow, C. Karen Liu, Steven L. Martin, Zoran Popović, Rajesh P. N. Rao, Aaron P. Shon, Jack M. Wang
|Style machines||Style IK||Learning biomechanics||Motion Composition|
|Shared Latent Gaussian Processes||Gaussian Process Dynamical Models||Style-Content Gaussian Processes||Active learning for mocap|
Non-rigid tracking and reconstruction
How do we perceive the 3D structure of a video sequence that contains moving people and objects? We have developed several techniques that address aspects of this problem, both addressing fundamental computer vision questions, and also creating a practical application for special effects. In one project, we have developed a method for determining the 3D shape and motion of a non-rigid object from raw video, without any prior knowledge about the deforming object. This is based on our work on probabilistic non-rigid structure-from-motion. We have developed a method for detailed reconstruction of a shaded 3D object from video, for diffusely-reflecting surfaces. We have developed interactive tracking (rotoscoping) techniques that are now in use in the special effects industry.
Collaborators: Aseem Agarwala, Christoph Bregler, Marcus Brubaker, Andrew Fitzgibbon, David J. Fleet, Pascal Fua, Shahram Izadi, Cem Keskin, Varun Ramakrishna, David H. Salesin, Steven M. Seitz, Jamie Shotton, Richard Stebbing, Jonathan Taylor, Lorenzo Torresani, Raquel Urtasun
|Non-rigid SFM||Automatic non-rigid shape from video||Rotoscoping||Kinematic person tracking|
|Physics-Based Person Tracking||Hand reconstruction|
Controllers for simulated locomotion
We are developing techniques for creating human and animal motor controllers that move in physically-realistic and expressive ways. Our work is inspired by insights from biology, robotics, and reinforcement learning.
Collaborators: Mazen Al Borno, Eugene Fiume, David J. Fleet, Martin de Lasa, Igor Mordatch, Jack M. Wang
|Prioritized Optimization||Optimizing Walking||Walking with Uncertainty||Feature-Based Controllers|
|Low-Dimensional Planning||Full-Body Spacetime||Rotational control|
Machine learning for geometry processing
Collaborators: Brett Allen, Henning Biermann, Brian Curless, Andrew Fitzgibbon, Thomas Funkhouser, Leo Guibas, Evangelos Kalogerakis, Vladimir G. Kim, Wilmot Li, Tianqiang Liu, James McCrae, Derek Nowrouzezahrai, Gerard Pons-Moll, Zoran Popović, Jamie Shotton, Patricio Simari, Karan Singh, Hao Su, Jonathan Taylor, Li Yi, Lexing Ying, Ersin Yumer, Denis Zorin
|Surface Texture Synthesis||Learning body shape variation||Real-Time Curvature||Learning mesh segmentation|
|Furniture style||Metric Regression Forests||Learning segmentation from scraping|
Rigid shape reconstruction
Collaborators: Aseem Agarwala, Michael F. Cohen, Alex Colburn, Brian Curless, Dan B Goldman, Steven M. Seitz, Adrien Treuille, Li Zhang
|Smooth surfaces from video||Example-based photometric stereo||Example-based multiview stereo||Scanning with varying BRDFs|
Other computer vision and image processing
Collaborators: Vladimir Bychkovsky, Frédo Durand, Alexei A. Efros, Ali Farhadi, Rob Fergus, William T. Freeman, James Hays, Matthew D. Hoffman, Hamid Izadinia, Ronnachai Jaroensri, Jiaya Jia, Evangelos Kalogerakis, Sylvain Paris, Brian Price, Sam T. Roweis, Bryan C. Russell, Ian Sachs, Eli Shechtman, Xiaoyong Shen, Barun Singh, Olga Vesselova
|Single-image deblurring||Image Sequence Geolocation||Acceptable photographic adjustments||Deep image tagging|
Other machine learning and data science
Collaborators: Kannan Achan, Marcus Brubaker, Yanshuai Cao, Mira Dontcheva, David J. Fleet, Brendan Frey, Michael Guerzhoy, Matthew D. Hoffman, Zhicheng Liu, Sam T. Roweis, Maja Rudolph, Alan Wilson, Jian Zhao
|Segmental speech processing||Latent Factor Travel Model||Sparse Gaussian Processes||Event sequence visualization|