Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics

June 2025   Bianchini

We published our project on building dynamics models from vision and contact-rich robot interaction at Robotics Science and Systems (RSS) 2025 in Los Angeles. My co-authors are Penn postdoc Minghan Zhu, our advisor Professor Michael Posa, and three other collaborators: Mengti Sun, Bowen Jiang, and Professor Camillo J Taylor. Our project presents Vysics, which is a vision-and-physics framework for a robot to build an expressive geometry and dynamics model of a single rigid body, using a seconds-long RGBD video and the robot's proprioception.

Video 1:  Vysics teaser video.

Our official webpage

Our official webpage is located here and embedded below for ease-of-viewing.

Paper abstract

We introduce Vysics, a vision-and-physics framework for a robot to build an expressive geometry and dynamics model of a single rigid body, using a seconds-long RGBD video and the robot's proprioception. While the computer vision community has built powerful visual 3D perception algorithms, cluttered environments with heavy occlusions can limit the visibility of objects of interest. However, observed motion of partially occluded objects can imply physical interactions took place, such as contact with a robot or the environment. These inferred contacts can supplement the visible geometry with “physible geometry,” which best explains the observed object motion through physics. Vysics uses a vision-based tracking and reconstruction method, BundleSDF, to estimate the trajectory and the visible geometry from an RGBD video, and an odometry-based model learning method, Physics Learning Library (PLL), to infer the “physible” geometry from the trajectory through implicit contact dynamics optimization. The visible and “physible” geometries jointly factor into optimizing a signed distance function (SDF) to represent the object shape. Vysics does not require pretraining, nor tactile or force sensors. Compared with vision-only methods, Vysics yields object models with higher geometric accuracy and better dynamics prediction in experiments where the object interacts with the robot and the environment under heavy occlusion.

Citation

@inproceedings{bianchini2025vysics,
  title={Vysics: Object Reconstruction Under Occlusion by Fusing Vision and Contact-Rich Physics},
  author={Bibit Bianchini* and Minghan Zhu* and Mengti Sun and Bowen Jiang and Camillo J. Taylor and Michael Posa},
  year={2025},
  month={june}, 
  booktitle={Robotics: Science and Systems (RSS)},
  website={https://vysics-vision-and-physics.github.io/}, 
}