The Defeat of the Winograd Schema Challenge

Our guest today is Vid Kocijan, a Machine Learning Engineer at Kumo AI. Vid has a Ph.D. in Computer Science at the University of Oxford. His research focused on common sense reasoning, pre-training in LLMs, pretraining in knowledge-based completion, and how these pre-trainings impact societal bias. He joins us to discuss how he built a BERT model that solved the Winograd Schema Challenge.

Vid introduced the Winograd schema challenge (WSC) and why it was challenging for deep learning models. He shared how he gathered the dataset he used for training the model. Vid also shared the metric for evaluating models for WSC. He shared the performance of other NLP algorithms on WSC.

Vid shared his thoughts on the imitation game and whether AI is advancing to pass the Turing test. He discussed other problems that still pose a challenge for AI systems.

Resources

Research Paper: A Surprisingly Robust Trick for Winograd Schema Challenge.

Research Paper: A Review of Winograd Schema Challenge Datasets and Approaches.