SE(3)-equivariant vision-based skill acquisition for robotic manipulation in contact-rich environments

ABOUT THIS PROJECT

At a glance

We propose to develop a new end-to-end learning framework for autonomous robotic assembly and construction tasks that require contact-rich interactions by learning from a small number of expert demonstrations. This will be accomplished by extending the application scope and capabilities of our recently developed Equivariant Descriptor Fields (EDFs) models: from learning end-to-end pick and place skills, utilizing visual perception, to learning end-to-end compliant assembly skills that encompass motion and force contact interaction primitives, and utilizing visual perception, internal and external force/torque measurements, and tactile sensing. Our initial task will be to seamlessly integrate recently developed bi-equivariant diffusion-EDFs’ vision-based task planning models with low-level equivariant geometric learning impedance controllers. Full SE(3) equivariance in both vision pose recognition and control forces can be achieved by combining these two approaches, resulting in efficient and robust learning of visual coordinate and force interaction skills to perform manipulation tasks in contact-rich environments.