Skip to the content Skip to the Navigation

Irfan Essa

  • Home
  • Blog
  • Publications
  • Team
  • Videos
  • Teaching
  • FAQ
  • Contact
Blog
  1. HOME
  2. Blog
  3. May 2022

May 2022

May 1, 2022 / Last updated : March 25, 2023 irfan ICRA

Paper in ICRA 2022 on “Graph-based Cluttered Scene Generation and Interactive Exploration using Deep Reinforcement Learning”

We introduce a novel method to teach a robotic agent to interactively explore cluttered yet structured scenes, such as kitchen pantries and grocery shelves, by leveraging the physical plausibility of the scene. We propose a novel learning framework to train an effective scene exploration policy to discover hidden objects with minimal interactions. First, we define a novel scene grammar to represent structured clutter. Then we train a Graph Neural Network (GNN) based Scene Generation agent using deep reinforcement learning (deep RL), to manipulate this Scene Grammar to create a diverse set of stable scenes, each containing multiple hidden objects. Given such cluttered scenes, we then train a Scene Exploration agent, using deep RL, to uncover hidden objects by interactively rearranging the scene. We show that our learned agents hide and discover significantly more objects than the baselines. We present quantitative results that prove the generalization capabilities of our agents. We also demonstrate sim-to-real transfer by successfully deploying the learned policy on a real UR10 robot to explore real-world cluttered scenes.

Recent Posts

Wizard of Oz at the Las Vegas Sphere, using Google AI
June 13, 2025
CVPR 2025 paper on “Cropper: Vision-Language Model for Image Cropping through In-Context Learning”
June 13, 2025
CVPR 2025 paper on “Calibrated Multi-Preference Optimization for Aligning Diffusion Models”
June 13, 2025
Award-winning paper in ICML 2024 on “VideoPoet: A large language model for zero-shot video generation.”
July 22, 2024
ACM SIGGRAPH Seminal Graphics Papers, Volume 2. Published as part of SIGGRAPH 50th Anniversary Meeting in 2023
August 9, 2023
Paper in UIST 2023 on “Slide Gestalt: Automatic Structure Extraction in Slide Decks for Non-Visual Access”
April 23, 2023
Award-winning paper in ICLR 2023 on “Emergence of Maps in the Memories of Blind Navigation Agents”
March 22, 2023
Paper in ICLR 2023 on “Discrete Predictor-Corrector Diffusion Models for Image Synthesis”
March 10, 2023
Some recent publications for 2023
March 10, 2023
Publications in 2022
December 31, 2022

Tags

ACM (20) Activity Recognition (52) Affective Computing (9) Aging-in-place (5) AI (20) Audio Analysis (9) Awards (15) Aware Home (15) Behavioral Imaging (11) Best Paper Award (11) Computational Journalism (36) Computational Photography (62) Computational Video (71) Computer Animation (10) Computer Graphics (9) Computer Vision (117) CVPR (30) DVFX (9) ECCV (5) Events (7) Faces (12) Funding (7) Generative Media (5) Gesture (6) Google (24) HCI (8) Health (7) ICCV (8) IEEE (30) Machine Learning (39) Medical (10) ML@GT (5) News (17) NSF (16) PhD Thesis (12) Presentations (28) Robotics (10) SIGGRAPH (7) Sports Visualization (6) Teaching (21) Ubiquitous Computing (5) Video Segmentation (7) Video Stabilization (14) WACV (8) Wearable Computing (9)

More about this Website

  • About
    • Tags & Categories
    • Archives
    • Copyright
    • Privacy Policy

Meta

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

Copyright © Irfan Essa All Rights Reserved.

Powered by WordPress with Lightning Theme & VK All in One Expansion Unit

MENU
  • Home
  • Blog
  • Publications
  • Team
  • Videos
  • Teaching
  • FAQ
  • Contact
PAGE TOP