Auditing LLMs and Twitter
Our guests, Erwan Le Merrer and Gilles Tredan, are long-time collaborators in graph theory and distributed systems. They share their expertise on applying graph-based approaches to understanding both large language model (LLM) hallucinations and shadow banning on social media platforms.
In this episode, listeners will learn how graph structures and metrics can reveal patterns in algorithmic behavior and platform moderation practices.
Key insights include the use of graph theory to evaluate LLM outputs, uncovering patterns in hallucinated graphs that might hint at the underlying structure and training data of the models, and applying epidemic models to analyze the uneven spread of shadow banning on Twitter.
Guests
Erwan Le Merrer: I am a researcher at Inria, Brittany, France (ARTISHAU team). I am leading the scientific council of the Société Informatique de France, and I am the president of an association promoting free software for mail/hosting in Brittany. My current research is auditing and evaluating black-box algorithms and AIs (or links to papers) in the context of of decision-making methods (neural-network models, LLMs, etc). In other words, I am interested in auditability, that is: what can or cannot be audited/evaluated, and how. I co-organized the WAAA workshop on algorithmic audits of algorithms, in an attempt to link several research domains on this societal question.
Gilles Tredan: Since November 2011 I work as a full time researcher in the TRUST (formerly TSF) group . I obtained a PhD degree in computer science from University of Rennes 1 in November 2009. From January 2010 to September 2011, I worked as a postdoc in the FG Inet group, Berlin. I defended my Habilitation a Diriger les Recherches in June 2019. My current focus is on algorithmic (adversarial) transparency: how to infer properties of remote (online) algorithms ? Which properties can be inferred at reasonable cost ? Can such approaches be used by societies to dispute with tech giants over the control of our digital existences ? I am coordinating the French ANR PACMAM project that explores audit-related problems. The objective of my thesis, entitled "Structures and Distributed systems", was to study the impact of communications structures on various distributed systems. My Habilitation a Diriger les Recherches focussed on the problem of graph "metrology". Graphs are widely used to model real-life homogeneous systems. However, accurately capturing interactions in such systems into a graph is a challenge. I thus explored techniques to evaluate and mitigate the impact of such inaccuracies.