Paper in CVPR 2019 on “Embodied Question Answering in Photorealistic Environments with Point Cloud Perception”

Abstract

To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task – Embodied Question Answering in photo-realistic environments (Matterport 3D). We thoroughly study navigation policies that utilize 3D point clouds, RGB images, or their combination. Our analysis of these models reveals several key findings. We find that two seemingly naive navigation baselines, forward-only and random, are strong navigators and challenging to outperform, due to the specific choice of the evaluation settings. We find a novel loss-weighting scheme we call Inflection Weighting to be important when training recurrent models for navigation with behavior cloning and are able to out-perform the baselines with this technique. We find that point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating the use (and continued study) of 3D deep learning models for embodied navigation.

Spotlight Presentation at CVPR 2019

Citation

  • E. Wijmans, S. Datta, O. Maksymets, A. Das, G. Gkioxari, S. Lee, I. Essa, D. Parikh, and D. Batra (2019), “Embodied Question Answering in Photorealistic Environments With Point Cloud Perception,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. [PDF] [VIDEO] [DOI] [arXiv] [BIBTEX]
    @InProceedings{ 2019-Wijmans-EQAPEWPCP,
    arxiv = {https://arxiv.org/abs/1904.03461},
    author  = {Wijmans, Erik and Datta, Samyak and Maksymets,
    Oleksandr and Das, Abhishek and Gkioxari, Georgia
    and Lee, Stefan and Essa, Irfan and Parikh, Devi and
    Batra, Dhruv},
    booktitle  = {{IEEE Conference on Computer Vision and Pattern
    Recognition (CVPR)}},
    doi = {10.1109/CVPR.2019.00682},
    month = {June},
    pdf = {http://openaccess.thecvf.com/content_CVPR_2019/papers/Wijmans_Embodied_Question_Answering_in_Photorealistic_Environments_With_Point_Cloud_Perception_CVPR_2019_paper.pdf},
    title = {Embodied Question Answering in Photorealistic
    Environments With Point Cloud Perception},
    video = {https://youtu.be/Je5LlZlqUt8?t=2483},
    year = {2019}
    }

More

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.