A searchable list of some of my publications is below. You can also access my publications from the following sites.
My ORCID is
https://orcid.org/0000-0002-6236-2969
Publications:
1.
S. Hickson, N. Dufour, A. Sud, V. Kwatra, I. Essa
Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras Proceedings Article
In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1626-1635, 2019, ISSN: 1550-5790.
Abstract | Links | BibTeX | Tags: audio-video fusion, face & gesture, face processing, multimodal interfaces, WACV
@inproceedings{2019-Hickson-ECFEUEC,
title = {Eyemotion: Classifying Facial Expressions in VR Using Eye-Tracking Cameras},
author = {S. Hickson and N. Dufour and A. Sud and V. Kwatra and I. Essa},
url = {https://ieeexplore.ieee.org/document/8658392
https://ai.google/research/pubs/pub46291},
doi = {10.1109/WACV.2019.00178},
issn = {1550-5790},
year = {2019},
date = {2019-01-01},
urldate = {2019-01-01},
booktitle = {IEEE Winter Conference on Applications of Computer Vision (WACV)},
pages = {1626-1635},
abstract = {One of the main challenges of social interaction in virtual reality settings is that head-mounted displays occlude a large portion of the face, blocking facial expressions and thereby restricting social engagement cues among users. We present an algorithm to automatically infer expressions by analyzing only a partially occluded face while the user is engaged in a virtual reality experience. Specifically, we show that images of the user's eyes captured from an IR gaze-tracking camera within a VR headset are sufficient to infer a subset of facial expressions without the use of any fixed external camera. Using these inferences, we can generate dynamic avatars in real-time which function as an expressive surrogate for the user. We propose a novel data collection pipeline as well as a novel approach for increasing CNN accuracy via personalization. Our results show a mean accuracy of 74% (F1 of 0.73) among 5 'emotive' expressions and a mean accuracy of 70% (F1 of 0.68) among 10 distinct facial action units, outperforming human raters.
},
keywords = {audio-video fusion, face & gesture, face processing, multimodal interfaces, WACV},
pubstate = {published},
tppubtype = {inproceedings}
}
One of the main challenges of social interaction in virtual reality settings is that head-mounted displays occlude a large portion of the face, blocking facial expressions and thereby restricting social engagement cues among users. We present an algorithm to automatically infer expressions by analyzing only a partially occluded face while the user is engaged in a virtual reality experience. Specifically, we show that images of the user's eyes captured from an IR gaze-tracking camera within a VR headset are sufficient to infer a subset of facial expressions without the use of any fixed external camera. Using these inferences, we can generate dynamic avatars in real-time which function as an expressive surrogate for the user. We propose a novel data collection pipeline as well as a novel approach for increasing CNN accuracy via personalization. Our results show a mean accuracy of 74% (F1 of 0.73) among 5 'emotive' expressions and a mean accuracy of 70% (F1 of 0.68) among 10 distinct facial action units, outperforming human raters.
2.
Sumit Basu, Irfan Essa, Alex Pentland
Motion Regularization for Model-based Head Tracking Proceedings Article
In: Proceedings of International Conference on Pattern Recognition (ICPR), 1996, ISBN: 0-8186-7282-X.
Abstract | Links | BibTeX | Tags: computer vision, face & gesture, ICPR
@inproceedings{1996-Basu-MRMHT,
title = {Motion Regularization for Model-based Head Tracking},
author = {Sumit Basu and Irfan Essa and Alex Pentland},
url = {https://ieeexplore.ieee.org/document/547019},
doi = {10.1109/ICPR.1996.547019},
isbn = {0-8186-7282-X},
year = {1996},
date = {1996-10-01},
urldate = {1996-10-01},
booktitle = {Proceedings of International Conference on Pattern Recognition (ICPR)},
abstract = {This paper describes a method for the robust tracking of rigid head motion from video. This method uses a 3D ellipsoidal model of the head and interprets the optical flow in terms of the possible rigid motions of the model. This method is robust to large angular and translational motions of the head and is not subject to the singularities of a 2D model. The method has been successfully applied to heads with a variety of shapes, hair styles, etc. This method also has the advantage of accurately capturing the 3D motion parameters of the head. This accuracy is shown through comparison with a ground truth synthetic sequence (a rendered 3D animation of a model head). In addition, the ellipsoidal model is robust to small variations in the initial fit, enabling the automation of the model initialization. Lastly, due to its consideration of the entire 3D aspect of the head, the tracking is very stable over a large number of frames. This robustness extends even to sequences with very low frame rates and noisy camera images.
},
keywords = {computer vision, face & gesture, ICPR},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper describes a method for the robust tracking of rigid head motion from video. This method uses a 3D ellipsoidal model of the head and interprets the optical flow in terms of the possible rigid motions of the model. This method is robust to large angular and translational motions of the head and is not subject to the singularities of a 2D model. The method has been successfully applied to heads with a variety of shapes, hair styles, etc. This method also has the advantage of accurately capturing the 3D motion parameters of the head. This accuracy is shown through comparison with a ground truth synthetic sequence (a rendered 3D animation of a model head). In addition, the ellipsoidal model is robust to small variations in the initial fit, enabling the automation of the model initialization. Lastly, due to its consideration of the entire 3D aspect of the head, the tracking is very stable over a large number of frames. This robustness extends even to sequences with very low frame rates and noisy camera images.
3.
Irfan Essa, Sumit Basu, Trevor Darrell, Alex Pentland
Modeling, Tracking and Interactive Animation of Faces and Heads using Input from Video Proceedings Article
In: Computer Animation Conference, pp. 68–79, IEEE Computer Society Press, 1996, ISBN: 0-8186-7588-8.
Abstract | Links | BibTeX | Tags: computer vision, face & gesture
@inproceedings{1996-Essa-MTIAFHUIFV,
title = {Modeling, Tracking and Interactive Animation of Faces and Heads using Input from Video},
author = {Irfan Essa and Sumit Basu and Trevor Darrell and Alex Pentland},
url = {https://ieeexplore.ieee.org/abstract/document/540489},
doi = {10.1109/CA.1996.540489},
isbn = {0-8186-7588-8},
year = {1996},
date = {1996-06-01},
urldate = {1996-06-01},
booktitle = {Computer Animation Conference},
pages = {68--79},
publisher = {IEEE Computer Society Press},
abstract = {We describe tools that use measurements from video for the extraction of facial modeling and animation parameters, head tracking, and real time interactive facial animation. These tools share common goals but rely on varying details of physical and geometric modeling and in their input measurement system. Accurate facial modeling involves fine details of geometry and muscle coarticulation. By coupling pixel by pixel measurements of surface motion to a physically based face model and a muscle control model, we have been able to obtain detailed spatio temporal records of both the displacement of each point on the facial surface and the muscle control required to produce the observed facial motion. We discuss the importance of this visually extracted representation in terms of realistic facial motion synthesis. A similar method that uses an ellipsoidal model of the head coupled with detailed estimates of visual …
},
keywords = {computer vision, face & gesture},
pubstate = {published},
tppubtype = {inproceedings}
}
We describe tools that use measurements from video for the extraction of facial modeling and animation parameters, head tracking, and real time interactive facial animation. These tools share common goals but rely on varying details of physical and geometric modeling and in their input measurement system. Accurate facial modeling involves fine details of geometry and muscle coarticulation. By coupling pixel by pixel measurements of surface motion to a physically based face model and a muscle control model, we have been able to obtain detailed spatio temporal records of both the displacement of each point on the facial surface and the muscle control required to produce the observed facial motion. We discuss the importance of this visually extracted representation in terms of realistic facial motion synthesis. A similar method that uses an ellipsoidal model of the head coupled with detailed estimates of visual …
4.
T. Darrell, I. Essa, A. Pentland
Correlation and Interpolation Networks for Real-Time Expression Analysis/Synthesis Proceedings Article
In: Tesauro, G., Touretzky, D. S., Leen, T. K. (Ed.): Advances in Neural Information Processing Systems (NeurIPS), MIT Press, 1994.
Abstract | Links | BibTeX | Tags: face & gesture, face processing, gesture recognition
@inproceedings{1994-Darrell-CINREA,
title = {Correlation and Interpolation Networks for Real-Time Expression Analysis/Synthesis},
author = {T. Darrell and I. Essa and A. Pentland},
editor = {G. Tesauro and D. S. Touretzky and T. K. Leen},
url = {https://papers.nips.cc/paper/999-correlation-and-interpolation-networks-for-real-time-expression-analysissynthesis},
year = {1994},
date = {1994-12-01},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
volume = {7},
publisher = {MIT Press},
abstract = {We describe a framework for real-time tracking of facial expressions
that uses neurally-inspired correlation and interpolation methods. A
distributed view-based representation is used to characterize facial state,
and is computed using a replicated correlation network. The ensemble
response of the set of view correlation scores is input to a network based
interpolation method, which maps perceptual state to motor control states
for a simulated 3-D face model. Activation levels of the motor state
correspond to muscle activations in an anatomically derived model. By
integrating fast and robust 2-D processing with 3-D models, we obtain a
system that is able to quickly track and interpret complex facial motions
in real-time.},
keywords = {face & gesture, face processing, gesture recognition},
pubstate = {published},
tppubtype = {inproceedings}
}
We describe a framework for real-time tracking of facial expressions
that uses neurally-inspired correlation and interpolation methods. A
distributed view-based representation is used to characterize facial state,
and is computed using a replicated correlation network. The ensemble
response of the set of view correlation scores is input to a network based
interpolation method, which maps perceptual state to motor control states
for a simulated 3-D face model. Activation levels of the motor state
correspond to muscle activations in an anatomically derived model. By
integrating fast and robust 2-D processing with 3-D models, we obtain a
system that is able to quickly track and interpret complex facial motions
in real-time.
that uses neurally-inspired correlation and interpolation methods. A
distributed view-based representation is used to characterize facial state,
and is computed using a replicated correlation network. The ensemble
response of the set of view correlation scores is input to a network based
interpolation method, which maps perceptual state to motor control states
for a simulated 3-D face model. Activation levels of the motor state
correspond to muscle activations in an anatomically derived model. By
integrating fast and robust 2-D processing with 3-D models, we obtain a
system that is able to quickly track and interpret complex facial motions
in real-time.
5.
I. Essa, A. Pentland
A Vision System for Observing and Extracting Facial Action Parameters Proceedings Article
In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 76–83, IEEE Computer Society, 1994.
Abstract | Links | BibTeX | Tags: computer vision, CVPR, face & gesture, face processing
@inproceedings{1994-Essa-VSOEFAP,
title = {A Vision System for Observing and Extracting Facial Action Parameters},
author = {I. Essa and A. Pentland},
url = {https://ieeexplore.ieee.org/document/323813},
doi = {10.1109/CVPR.1994.323813},
year = {1994},
date = {1994-01-01},
urldate = {1994-01-01},
booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {76--83},
publisher = {IEEE Computer Society},
abstract = {We describe a computer vision system for observing the "action units" of a face using video sequences as input. The visual observation (sensing) is achieved by using an optimal estimation optical flow method coupled with a geometric and a physical (muscle) model describing the facial structure. This modeling results in a time-varying spatial patterning of facial shape and a parametric representation of the independent muscle action groups, responsible for the observed facial motions. These muscle action patterns may then be used for analysis, interpretation, and synthesis. Thus, by interpreting facial motions within a physics-based optimal estimation framework, a new control model of facial movement is developed. The newly extracted action units (which we name "FACS+") are both physics and geometry-based, and extend the well-known FACS parameters for facial expressions by adding temporal information and non-local spatial patterning of facial motion.< >
},
keywords = {computer vision, CVPR, face & gesture, face processing},
pubstate = {published},
tppubtype = {inproceedings}
}
We describe a computer vision system for observing the "action units" of a face using video sequences as input. The visual observation (sensing) is achieved by using an optimal estimation optical flow method coupled with a geometric and a physical (muscle) model describing the facial structure. This modeling results in a time-varying spatial patterning of facial shape and a parametric representation of the independent muscle action groups, responsible for the observed facial motions. These muscle action patterns may then be used for analysis, interpretation, and synthesis. Thus, by interpreting facial motions within a physics-based optimal estimation framework, a new control model of facial movement is developed. The newly extracted action units (which we name "FACS+") are both physics and geometry-based, and extend the well-known FACS parameters for facial expressions by adding temporal information and non-local spatial patterning of facial motion.< >
Other Publication Sites
A few more sites that aggregate research publications: Academic.edu, Bibsonomy, CiteULike, Mendeley.
Copyright/About
[Please see the Copyright Statement that may apply to the content listed here.]
This list of publications is produced by using the teachPress plugin for WordPress.