AI Fails on Theory of Mind Tasks
On the show today, we are joined by Tomer Ullman, an assistant professor of Psychology at Harvard University. He is a cognitive scientist with research in cognitive modeling and development.
Tomer started by sharing the intersection between psychology and AI. He shared his thoughts on whether AI models can be developed without knowledge and learn as they experience the world. Tomer gave a rich explanation of the theory of mind and whether machines now possess the property. He explained how the Sally-Anne test measures a machine’s level of the theory of mind.
He also shared his thoughts about animals having the theory of mind. He discussed the evolution of published datasets to evaluate machines for the theory of mind test. Tomer shared how he conducted his experiment that shows LLMs fail the theory of mind tests, a study he published. He explained the Smarties tube test — another test for theory of mind. He also discussed how he made variations of the Sally-Anne test and the Smarties tube test on GPT 3.5 to test for the theory of mind.
Tomer shared his views on developing more advanced tests for the theory of mind in machines. To learn more about Tomer’s work, follow him on Twitter @TomerUllman or check his website.
Tomer Ullman
Tomer Ullman is a cognitive scientist interested in common-sense reasoning, and building computational models for explaining high-level cognitive processes and the acquisition of new knowledge by children and adults. In particular, he is focused on how children and adults come to form intuitive theories of agents and objects, and providing both a functional and algorithmic account of how these theories are learned. Such an account would go a long way towards explaining the basics cogs and springs of human intelligence, and support the building of more human-like artificial intelligence. Dr. Ullman received in B.Sc in Cognitive Science and Physics from Hebrew University in 2008, and his Ph.D. in Brain and Cognitive Sciences from MIT in 2015. From 2015-2018 he was a post-doctoral associate at the Center for Brains, Minds, and Machines.
