Tentative schedule
Welcome
Talk
Embodied Multimodal Intelligence with Foundation Models
Oier Mees (UC Berkeley)
Despite considerable progress in robot learning and contrary to the expectations of the general public, the vast majority of robots deployed out in the real world today continue to remain restricted to a narrow set of preprogrammed behaviors for specific tasks. As robots become ubiquitous across human-centred environments, the need for "generalist" robots grows: how can we scale robot learning systems to generalize and adapt, allowing them to perform a wide range of everyday tasks in unstructured environments based on arbitrary, multimodal instructions from the users? In my work, I have focused on addressing the challenging problems of relating human language to a robot's multimodal perceptions and actions by introducing techniques that leverage self-supervision from uncurated data and common sense reasoning from foundation models from and for robotics.
Talk
Systems 1 and 2 for Robot Learning
Katerina Fragkiadaki (CMU)
Humans can successfully handle both mundane and new and rare tasks simply by thinking harder and being more focused. How can we develop robots that think harder and do better in out-of-distribution scenarios? In this talk, we will marry today's generative models and traditional evolutionary search to enable better generalization of robot policies, and the ability to test-time reason through difficult scenarios, akin to a robot system 2. We will discuss learning behaviours from videos with 3D video perception as well as through language instructions and corrections that shape the robots' reward functions on-the-fly, and help us automate robot training data collection in simulators and in the real world. We will also discuss the development of better and faster simulators as universal data engines for robotics.
Lunch Break
Poster Session
Robot Learning at KIT
Talk
Safe and Robust Real-World Robot Learning
Davide Tateo (TU Darmstadt)
Nowadays, it is clear that we need to incorporate learning methods to develop the robots of the future. However, learning is not enough to empower the robot to deal with challenging real-world scenarios: we need to enable the robots to adapt dynamically to the environment with online learning. To allow online learning in real robotic systems, we need to solve three key challenges: efficient learning, robustness to disturbances, and satisfaction of safety constraints. In this talk, we will discuss these challenges and show how to deploy learning methods in complex contact-rich dynamic tasks such as the robot Air Hockey setting.
Final Remarks
Speakers
Katerina Fragkiadaki
Katerina Fragkiadaki is the JPMorgan Chase Associate Professor in the Machine Learning Department in Carnegie Mellon University. She received her undergraduate diploma from Electrical and Computer Engineering in the National Technical University of Athens. She received her Ph.D. from University of Pennsylvania and was a postdoctoral fellow in UC Berkeley and Google research after that. Her work focuses on combining forms of common sense reasoning, such as spatial understanding and 3D scene understanding, with deep visuomotor learning. The goal of her work is to enable few-shot learning and continual learning for perception, action and language grounding. Her group develops methods for computer vision for mobile agents, 2D and 3D visual parsing, 2D-to-3D perception, vision-language grounding, learning of object dynamics, navigation and manipulation policies. Pioneering innovations of her group's research include 2D-to-3D geometry-aware neural networks for 3D understanding from 2D video streams, analogy-forming networks for memory-augmented few-shot visual parsing, and language-grounding in 2D and 3D scenes with bottom-up and top-down attention. Her work has been awarded with a best Ph.D. thesis award, an NSF CAREER award, AFOSR Young Investigator award, a DARPA Young Investigator award, Google, TRI, Amazon, UPMC and Sony faculty research awards. She is a program chair for ICLR 2024.
Davide Tateo
Davide Tateo is a Research Group Leader at the Intelligent Autonomous Systems Laboratory in the Computer Science Department of the Technical University of Darmstadt. He received his M.Sc. degree in Computer Engineering at Politecnico di Milano in 2014 and his Ph.D. in Information Technology from the same university in 2019. Davide Tateo worked in many areas of Robotics and Reinforcement Learning, Planning, and Perception. His main research interest is Robot Learning, focusing on high-speed motions, locomotion, and safety.
Oier Mees
Oier Mees is a PostDoc at UC Berkeley working with Prof. Sergey Levine. He received his PhD in Computer Science (summa cum laude) in 2023 from the Freiburg University supervised by Prof. Dr. Wolfram Burgard. His research focuses on robot learning, with the goal of enabling robots to intelligently interact with both the physical world and humans, and improve themselves over time. Concretely, he is interested in how we can build self-improving embodied foundation models that can generalize the same way humans do. His research has been nominated for (and received) several Best Paper Awards, including ICRA and RA-L. Previously, he also spent time at NVIDIA AI interning with Dieter Fox.