Talk Abstract
Large language models have revolutionized textual reasoning, yet their ability to act meaningfully in multimodal, real-world environments remains limited. They struggle to ground their decisions in visual context, adapt to changing goals, and plan actions over time—shortcomings that stem from a lack of structured, goal-driven reasoning and insufficient representations of the physical world.
In this talk, I present a unified framework for building embodied agents that can see, simulate, and reason. I begin by introducing methods for learning world simulators from data, arguing that visual reasoning—like textual reasoning—benefits from step-by-step processing. Inverting a physics simulator becomes key: agents must infer structured 3D neural representations of objects, parts, motions, and scenes directly from raw video. I describe methods for extracting such representations using generative priors, injecting them into vision-language models (VLMs), and scaling up their supervision via generative 3D simulation and fast, modular physics engines. These simulators enable agents to anchor their predictions in grounded physical reality, reducing hallucinations and improving control.
Complementing this simulation capability, I explore techniques that enable agents to reason over time and adapt their behavior. By integrating structured memory systems, agents learn to retain and retrieve relevant experiences to inform long-horizon plans. Language-based reflective feedback allows them to refine their strategies beyond what sparse rewards offer, forming abstractions that generalize across tasks. When trained to ground their reasoning directly in the visual environment, agents gain the ability to set subgoals, explore effectively, and verify their own hypotheses.
Together, these advances point toward autonomous systems that simulate before they act, reflect after they fail, and maintain an ongoing awareness of goals, constraints, and context. I will illustrate these capabilities across web automation, robotics, and interactive assistance, showing how agents that see, simulate, and reason offer a promising path toward general-purpose embodied intelligence.
Biography
Katerina Fragkiadaki is the JPMorgan Chase Associate Professor in the Machine Learning Department at Carnegie Mellon University. She received her undergraduate degree in Electrical and Computer Engineering from the National Technical University of Athens, and her Ph.D. from the University of Pennsylvania. She subsequently held postdoctoral positions at UC Berkeley and Google Research. Her research focuses on enabling few-shot and continual learning for perception, action, and language grounding. Her work has been recognizedby the Best Ph.D. Thesis Award, NSF CAREER Award, AFOSR and DARPA Young Investigator Awards, as well as faculty research awards from Google, Toyota Research Institute, Amazon, NVIDIA, UPMC, and Sony. She served as a Program Chair for ICLR 2024.
Location
Sennott Square, Room 5317
Date
Friday, Jan. 9 at noon to 2 p.m.