Broadly speaking, I am interested in all areas of computer graphics and computer vision. My work seeks to address the following high-level questions:
Computer Graphics: What powerful tools can we provide to artists, designers, scientists, and novice users for creating beautiful, expressive, artistic, and/or illustrative imagery and animation?
Computer Vision: How can we visually understand the world, extract meaning from images, and model the human visual system?
Moreover, I am increasingly interested in applications of Machine Learning — especially Bayesian inference — to these two areas. Computer vision and graphics both rely heavily on analysis and generation of data, and Bayesian learning provides extremely powerful tools for interpreting data. I collaborate actively with my colleagues at UofT. UofT has some of the strongest groups around in the areas of computer vision, machine learning, and graphics/HCI. I also collaborate with a number of labs internationally, including University of Washington's GRAIL lab.
Some specific research areas:
Character animation by example
How can we create virtual
characters from live human performance? Doing so requires building computational models of human motion that incorporate physics of the body as well as an individual's style, explicitly or implicitly.
Our work in this area led to
the first system for creating animation
by example, an area that has since become very popular in the
graphics community. We have developed
a real-time
system for interactive
character posing that uses machine learning to help determine
which poses are "most likely," and has been licensed to a major game
developer. More recently, we have developed some exciting new methods
for learning biomechanical human body models.
This method can predict how a person will move under
new circumstances, based on muscle strengths and stiffnesses estimated
from a motion capture sequence.
Collaborators: Brett Allen, Matthew Brand, Brian Curless, Keith Grochow, C. Karen Liu, Steven L. Martin, Zoran Popović
|
|
|
|
| Style machines | Style IK | Learning biomechanics | Complex Motion Composition |
|
|
||
| Learning body shape variation | Active learning for mocap |
Visual tracking, reconstruction, and
rotoscoping
How do we perceive the 3D structure of a video sequence that contains
moving people and objects? We have developed several techniques that
address aspects of this problem, both addressing fundamental computer
vision questions, and also creating a practical application for
special effects. In one project, we have developed a method for
determining the 3D shape and motion of a non-rigid object from raw
video, without any prior knowledge about the deforming object.
This is based on our work on probabilistic non-rigid
structure-from-motion.
We have developed a method for detailed reconstruction of a shaded 3D object from video, for diffusely-reflecting surfaces.
We
are currently working on different methods for tracking human motion
in video sequences. We have
developed interactive tracking (rotoscoping) techniques that are now
in use in the special effects industry.
Collaborators: Aseem Agarwala, Chris Bregler, Marcus Brubaker, David Fleet, Pascal Fua, David Salesin, Steve Seitz, Lorenzo Torresani, Raquel Urtasun, Li Zhang
|
|
|
|
| Automatic non-rigid shape from video | Non-rigid SFM | Smooth surfaces from video | Rotoscoping |
|
|
||
|
|
Physics-Based Person Tracking |
Image processing and texture synthesis by example
How can we creating sophisticated image filters without a lot of
difficult programming? We have developed a technique called
Image
Analogies that
learns transformations from user-provided examples.
Within
a single learning framework, we can learn many different types of
transformations, including generating new texture, generating new
landscapes, and learning the styles of famous artists like Van Gogh
and
Manet.
We have also developed methods for
generating textures on 3D surfaces.
Collaborators: Henning Biermann, Brian Curless, Chuck Jacobs, Nuria Oliver, David Salesin, Lexing Ying, Denis Zorin
![]() |
|
| Image Analogies | Surface texture synthesis |
Reconstructing objects with real-world materials
We have developed methods for 3D shape reconstruction of static
objects in the difficult case of complex reflectance (such as shiny
objects), cases that foil most shape reconstruction methods, including
laser scanners. We have developed a
method based on photometric stereo using reference objects that
entails very simple setup and calibration, and does not require
advance knowledge of shape, illumination, or materials.
We have developed a more recent technique that reconstructs BRDFs as
well and can be used for relighting and rerendering.
Collaborators: Brian Curless, Dan Goldman, Steve Seitz, Adrien Treuille
|
|
|
| Example-based photometric stereo | Example-based multiview stereo | Scanning with varying BRDFs |
Non-photorealistic rendering and automated design
How can we write computer software that helps in creating
artistic imagery and video? How can we enable computer animation in
the styles of human painting and drawing? Answering these questions
involves understanding aspects of artistic style and human
perception. We have developed technqiues for
painterly
rendering,
painterly
animation, and pen-and-ink illustration of 3D
surfaces.
Collaborators: Todd Goodwin, Alex Kolliopoulos, Ken Perlin, Ian Vollick, Jack Wang, Denis Zorin
|
|
|
|
| Painterly rendering | Painterly video | Pen-and-ink illustration | Segmentation-Based NPR |
|
|||
| Artistic Stroke Thickness |
Information visualization
Collaborators: Maneesh Agrawala, Daniel Vogel, Ian Vollick
|
| Learning Label Layout |
Visual scene inference
We are investigating Bayesian methods for interpreting the structure and
contents of one or two photographs, separating effects of shape,
lighting, and texture.
Collaborators: Bill Freeman, Rob Fergus, Sam Roweis, Barun Singh
|
| Removing Camera Shake |
Machine learning algorithms and applications
We have developed core learning algorithms and applications of learning to various applications, including speech processing, robotic imitation, and time-series analysis. Our techniques typically build on probabilistic inference techniques, such as latent variable models and Gaussian Processes.
Collaborators: Kannan Achan, David Fleet, Brendan Frey, Aaron Shon, Raj Rao, Sam Roweis, Jack Wang
|
|
|
|
| Gaussian Process Dynamical Models | Segmental speech processing | Learning Latent Structure for Image Synthesis and Robotic Imitation | Style-Content Gaussian Processes |
Learning controllers and behaviors
We are beginning to investigate techniques for creating human and
animal motor controllers that move in physically-realistic and
expressive ways. Our work is inspired by insights from biology,
robotics, and reinforcement learning.