Siddharth Karamcheti (Stanford)- Language-Drive Learning for Interactive Robotics

Abstract: Building and deploying broadly capable robots requires systems that can efficiently learn from and work with people. To achieve this, robots must balance capability — the fundamental tools necessary to enable real-world deployment — and sustainability — the ability to grow and adapt through human feedback. In this talk, I will motivate language-driven learning to tackle these axes, providing robots with better abstractions for perception, action, and human-robot interaction. Towards capability, I will present Voltron, our approach for using language to learn visual representations that can be efficiently adapted for diverse robotics tasks. Building on these ideas, I will discuss Prismatic, our experimental framework for developing visually-conditioned language models and vision-language-action policies at scale. Towards sustainability, I will next introduce Vocal Sandbox, a new framework that integrates these models to develop collaborative robots that can work alongside human partners, using language to express uncertainty and learn new behaviors from real-time interactions. Finally, I will conclude with open challenges for enhancing both the capability and sustainability of modern robots, with directions for future work.