Automated Peer Review

In this episode, we are joined by Ryan Liu, a Computer Science graduate of Carnegie Mellon University. Ryan will begin his Ph.D. program at Princeton University this fall. His Ph.D. will focus on the intersection of large language models and how humans think. Ryan joins us to discuss his research titled: ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

Ryan started by providing an overview of the peer review process. He discussed what birthed the idea of using LLMs, such as ChatGPT, Bard, LLaMa etc., for paper review. He discussed the approach used for the study and the kind of errors the LLMs made during evaluation.

Ryan shared the prompts he used for evaluating the papers for various use cases. Rounding up, he shared future use cases of using LLMs for paper reviews. He also shared details about his ongoing research.

Learn more about Ryan’s work on his Google Scholar page.