Music Playlist Recommendations
In this episode, Rebecca Salganik, a PhD student at the University of Rochester with a background in vocal performance and composition, discusses her research on fairness in music recommendation systems. She explores three key types of fairness—group, individual, and counterfactual—and examines how algorithms create challenges like popularity bias (favoring mainstream content) and multi-interest bias (underserving users with diverse tastes). Rebecca introduces LARP, her multi-stage multimodal framework for playlist continuation that uses contrastive learning to align text and audio representations, learn song relationships, and create playlist-level embeddings to address the cold start problem. A significant contribution of Rebecca's work is the Music Semantics dataset, created by scraping Reddit discussions to capture how people naturally describe music using atmospheric qualities, contextual comparisons, and situational associations rather than just technical features. This dataset, available on Hugging Face, enables more nuanced recommendation systems that better understand user preferences and support niche tastes. Her research utilizes industry datasets including Last.fm and Spotify's Million Playlist Dataset, and points toward exciting future applications in music generation and multimodal systems that combine audio, text, and video.
Guest
Rebecca Salganik: Rebecca Salganik is a computer scientist and musician whose research bridges artificial intelligence, fairness, and music discovery. She is currently pursuing a PhD in Computer Science at the University of Rochester in New York under Prof. Jian Kang, focusing on Individual Fairness in Graphs and Discovery in Recommender Systems. She completed her M.Sc. at Université de Montréal (MILA), where her thesis on fairness-aware music recommendation explored how to mitigate popularity bias in algorithmic systems—read it here. Her research, published at top conferences including ECIR, KDD, FAccT, and RecSys, examines how Graph Neural Networks, Fairness, and Music Recommender Systems can be designed to amplify diverse voices and create more equitable pathways for artistic expression. Rebecca has held research positions at Pinterest, Pandora Labs, Lyft, and MIT Lincoln Labs, and she organizes DEFirst, a reading group focused on fairness in information retrieval. Her background as a classically trained vocalist from McGill University deeply informs her interdisciplinary approach to building ethical AI systems that respect the nuances of musical expression.
Transcript
Kyle Polich : Welcome to Data Skeptic a podcast exploring the methods use cases and consequences of recommender systems
Kyle Polich : Welcome to another installment of Data skeptic recommender Systems Today we're focused on one of my favorite topics music or more specifically music recommendations in the form of playlist generation Our guest today Rebecca has an excellent background We start off the discussion talking mostly about fairness because I got curious when I learned about her work in this area but eventually we'll get around to talking about LARP as well as music SEM
Kyle Polich : That's a novel framework and an outstanding research data set available for work in this area We cover a lot of ground so I'm gonna cut the intro short and jump right into our discussion
Rebecca Salganik : My name's Rebecca Salganik and I'm affiliated with the University of Rochester where I'm co-supervised by John King and Anson Kong So I'm in the 2nd year of my PhD and I work on fairness in general and then I apply it often to graphs recommender systems and more recently dynamic systems and computational social choice So
Kyle Polich : I know you've done a lot of applications in music How did you get started applying your craft and recommenders to music specifically
Rebecca Salganik : So actually I started with an interest in music So a lot of the work that I do is applied to music because it's both an industry and academic kind of like open research question
Rebecca Salganik : How do we define similarity in music How do we apply recommender systems to music So I had a background in music I went to conservatory I did a classical degree in voice and composition and while I was there I started taking computer science classes I was working as a songwriter and one of the things that I was interested in is OK there's all these recommender systems and they are essentially becoming the gatekeepers for how music is created consumed and commodified
Rebecca Salganik : How can I learn about these systems to like help my music get discovered and that kind of transitioned into research in recommender systems which then also kind of transitioned into research on fairness because they started to see a lot of the problems that I had been facing were actually byproducts of certain aspects of recommender system design and so fairness is a tool for addressing this misalignment between what we want recommender systems to do
Rebecca Salganik : Which is kind of help people discover new things and what they currently do which is often um perpetuate mainstream patterns in people's consumption habits right So fairness is in the real world Fairness is an experience that someone has It's very personalized and so it's kind of nebulous but if we want to apply it to machine learning then what we want to do is basically be able to formalize it with respect to some sort of variables and mathematical functions
Rebecca Salganik : So that we can assess the relationship between the different outputs of an algorithmic system So there are generally 3 different branches of fairness and they're basically defined based on what kind of information you're going to use when you're talking about fairness So the most kind of canonical one is called group fairness and that has a basis in kind of like our legal system So it's the idea of like protected groups and sensitive attributes We can talk about like race
Rebecca Salganik : gender ethnicity socioeconomic background and so if you have the presence of these uh demographic information then what you can do is you can look for equality You can say when you have an algorithmic system that interacts with someone from a protected group and a non-protected group what's the difference between the kinds of outcomes it produces That's kind of the most canonical one The one that I focus most on is called individual fairness and the idea here
Rebecca Salganik : Is that an algorithmic system should treat similar individuals similarly And we can dive into what that means but I'm just going to kind of cover the third one which is called counterfactual fairness So what we know is that like socio-demographics kind of our protected attributes are not the only information that we need when we try to equalize for fairness right So there can be confounding factors that affect somebody For example if we're talking about
Rebecca Salganik : Ranking candidates for a job that kind of barred them from being able to achieve the same kinds of qualifications that another person could but that doesn't mean that they are less qualified So what it tries to do is it tries to understand how different variables affect each other and try to kind of de-noise the information that might be causing bias
Kyle Polich : So fairness is uh at least in my survey a relatively new introduction when applied to recommenders How did you first take an interest in this intersection
Rebecca Salganik : So the reason that I personally find individual fairness the most compelling is because I think what I'm trying to do in my work is to understand like this experience of fairness right When we experience unfairness we see a person who looks like us or is similar
Rebecca Salganik : To us in some way get something that we want or achieve an algorithmic outcome that we would have desired that we ourselves are not able to achieve And so the way that individual fairness works is given some information about individuals we can define similarity between them
Rebecca Salganik : And then what we can do is we can say OK we have this information about which individuals are similar and we also have information about their algorithmic outcomes How different are those And so sneak preview for recommender systems where we do talk a lot about similarity um individual fairness ends up playing a really important
Rebecca Salganik : role
Kyle Polich : If I were going to come at this from a naive perspective let's say I brought my knowledge of machine learning to the table but I didn't know much about music and for some reason I decided to take on music recommendations A first step in any AL project is faturization How do we take our raw data and make some features out of it that we can learn from and use Maybe step one would be to get the key of the music or the tempo or things like that but
Kyle Polich : That's probably a little bit too low level for describing what people actually care about in the music Ultimately it's like the creativity of music that the people prefer and are kind of after What are your thoughts on how you approach capturing that in some mathematical way
Rebecca Salganik : So I can give you my take but there is no one true answer right So if we want to backtrack and say what we want recommender systems to do the answer is that different people want recommender systems to do different things So people have different objectives and there are certainly many mainstream listeners who want to press play and have pop music come out of their system right But then there are also people who um have more needs
Rebecca Salganik : Listening tastes and that's actually the focus of my research is being able to find recommender systems that can modulate between the two because I would say that from my understanding we've currently done a really great job in the last 10 years of designing systems that can serve mainstream tastes but people who have diverse and niche tastes are often less represented in algorithmic design
Rebecca Salganik : So if you're asking me what should a recommender system be thinking about So in general in the recommendation space we have two different kinds of information that we can give it We have content which is what you were suggesting right with the kind of theory of the music The reason it's called content-based representation is because the only thing you care about is like the
Rebecca Salganik : Song itself the content you have And then we have context-based representation which is often also called collaborative filtering where you actually don't care what the song sounds like What you really care about is what people think like songs are similar So for example user one listened to these three songs together in one playlist
Rebecca Salganik : That must mean that these songs are similar So these are kind of the two perspectives Right now if we look at what's happening in the industry most of them are hybrid right So they're kind of mixing between the two but how to mix between the two what information we should be giving is kind of like the core of recommendation research in the music domain
Kyle Polich : Given your expertise in this area could you overview what it would be for a recommender system to be unfair in some way
Rebecca Salganik : So the there's many different ways I'm gonna start with the one that I work on the most So that's what we call like producer fairness And so that's thinking about like exposure that you give to artists So I mean societally we understand that when people listen to music artists get paid If you do not listen to their music they
Rebecca Salganik : They will not get paid They will not make more music right So for independent artists for niche artists who have less listeners it is absolutely an economic question um whether or not they will be able to make more music in the future and the algorithm's decision of recommending their songs or not basically gatekeeps whether they stay musicians or not
Rebecca Salganik : So that's the kind of unfairness that I think about Now what we have are different biases that are built into the algorithm because of the trading data that we give it and popularity bias is the one that I focus on the most And as you can intuit from the name here what the algorithm does is that it is biased towards popular content It is it is biased against unpopular content which could be niche content but it could also be new content
Rebecca Salganik : So a new artist that that that places their music on the platform Another recent work that I've been really interested in is users So one of the things that a recommender system does when it tries to make a recommendation is it tries to unify its understanding of a user using what's called an embedding right So this is just a representation of the user how it understands your taste But the thing is when it does that is it's
Rebecca Salganik : Only got one understanding And if you have a very very broad and diverse range of tastes versus another user who's kind of whose tastes are kind of like simple to capture so you will you have an equal amount of space that you can use to represent those two users And so there are recent findings that show that users who have diverse consumption patterns um are less well represented by algorithmic systems So this is another form of
Rebecca Salganik : I guess it would be called multi-interest bias
Kyle Polich : Well the problem we see in every recommender system playlist recommendation is no exception is the cold start problem For a new user you don't know much about them yet They haven't revealed any useful signals yet You've got to do something for them to get it going I guess we could design like a one size fits all model that kind of works for most people and that it would get us somewhere but I would think we could do better than that How do you go about it
Rebecca Salganik : One of the problems with popularity bias as I was mentioning before is that oftentimes an algorithmic system needs contextual information right so collaborative information in order to represent an item So we need someone to have already listened to that song we need that song to be placed in the playlist or interacted with to some degree before the algorithm
Rebecca Salganik : Can use content based representation but the problem is that when we have a new user or we have a new song or a new artist that is put onto the platform you cannot use context-based representation because it doesn't have enough interactions for us to use that methodology So we have to go back and we have to rely on content-based representation So this is called the cold start problem
Rebecca Salganik : Happens not just in like music recommendation it happens all over in other forms of recommendation e-commerce um and whatnot So one of the things that we have to do is we have to find a way to represent items even though we don't have interaction patterns for them And so that's what I worked on in LAARP
Kyle Polich : So LARP is a multi-stage framework Could you walk us through each stage
Rebecca Salganik : So in general the way that LARP works is that in the past content-based representation methods as we mentioned in like the earlier part of our conversation were really only using the music So information about the acoustics was kind of what we were using to define similarity between songs But what we tried to do in LAARP is to make this what's called multimodal So we want not just
Rebecca Salganik : The audio but also the text to be able to represent the song And so in order to achieve this what you need to do is called multimodal alignment So let's say you have a machine learning model that reasons about text You have a machine learning model that reasons about audio How do you combine the two such that they're in the same representational space This is an important question for being able to represent a song
Rebecca Salganik : And before I dive into more to that let me give some precursor The way in general that recommender systems work is they define an embedding space where they put all of the songs like the entire song catalog into this embedding space so that when it's time and this embedding space takes a long time to model and a long time to
Rebecca Salganik : Build And when it's time to serve recommendations like you're clicking next on a radio and it's like suggesting the next song all of that has been pre-computed before And what the recommender system is really doing is it's just going into that embedding space and it's looking for things that are close together in that embedding space What we can say is we've defined a mathematical function of distance
Rebecca Salganik : So like we're close together and that is a proxy for similarity in this space But how you build this space how you organize items in that space that's the key question of multimodal learning And that's the real question of recommendation systems because once you've constructed this space bringing stuff out of it like making the recommendations themselves is not the complicated part
Kyle Polich : So there's lots you could embed you have different forms of data about music What do you actually use in your case
Rebecca Salganik : We have this model that has a space that that reasons about text So you give it a song and it puts that song in the space It represents it in that space And you have the same thing for audio but the problem is that these two spaces are different What we need to do is kind of make sure that those spaces are aligned So the way that our framework works is it's used something called contrastive learning
Rebecca Salganik : So what it tries to do is it tries to first teach the algorithm just to align on the basis of one song So I give you the text of that song and I give you the audio of the song and the algorithm needs to learn that this is the same song so that when it embeds it in the space that's the correct place and all of the embedding is aligned so the text and the audio are representative together
Rebecca Salganik : But then when we think about how people try to make judgments about similarity between songs they're not just thinking about like the individual elements of that song they're also thinking about that song in relation to another song So what we do is we sample another track Let's say we have two songs in a playlist we will sample
Rebecca Salganik : Another song in the same playlist and we will try to align on the basis of those So now we'll talk about the text from one song and the audio from the other song the text from the second song and the audio from the first song and we try to reason about where in space they should be right Ideally they should be close together
Rebecca Salganik : And on the last stage what we want to now do is we want to zoom out and get more abstract So in the playlist continuation space right where we're trying to help a user build out a playlist what we need to do is we also need to make sure that all of the songs that are in a playlist are close together in this space So again when the recommender system is trying to help you build out your playlist it can go into that space and find the other songs that are close together with the ones you've already suggested
Rebecca Salganik : And so what we do is now alignment on the basis of the songs and also
Rebecca Salganik : the playlist
Kyle Polich : In terms of the text you can use is that lyrics or what sort of textual data do you have
Rebecca Salganik : In our work we just work with what's called the metadata We don't work with the lyrics um although you could use our framework for that The data sets that contain lyrics are often harder to get access to when you work in academia So what we work with is just the description so maybe the playlist name the song name the artist's name the genre that the artist participates in and also the album that kind of information
Kyle Polich : So creating embeddings for your features is a pretty popular approach and for good reason but those embeddings are embedding something some like kind of raw data or perhaps raw features Can you talk a little bit about your starting point and the inputs to the embedding How do you represent my music taste as a vector
Rebecca Salganik : Right so just to be clear here what we're trying to do is not necessarily embed your music preference but what we're trying to do is embed a song in relation to other songs So when we represent you we might use a different methodology but in this particular case we've got new songs or new users We don't know what your preferences are yet so we are forced to work only with the information that is available to us from kind of the intake
Rebecca Salganik : So when you're asking like what is the input to the framework great question So we have some songs and we have a text associated with each song So for example this song is uh Viva La Vida by Coldplay It has the following genres associated with it and it was in this this following year and it has it's part of the following album And then we also have the audio like the MP3 file the wave form associated with
Rebecca Salganik : Viva la vida And then we have the playlist that this particular song was in so maybe songs by Coldplay And we have all the other songs that were in that playlist And so when we train the first stage the input that we give to our algorithm is going to be just the text and the audio of the first song Viva La Vida by Coldplay and it's audio When we train the second stage we have two songs maybe Viva la Vida and then also Yellow
Rebecca Salganik : Right Because we're looking at songs by Coldplay and we have their textual annotation and we also have their audio And then for the last one we'll have the playlist So what we'll do is we'll take the 1st 10 songs of that playlist represent them as individual songs right We'll pass them through the network and we'll get like their text and their audio and their unified embedding and then we'll average them together And that's going to be the playlist representation And we're going to pass that into the 3rd stage of the algorithm
Kyle Polich : So I probably should have asked this earlier but all these multi-stages they all make sense towards an end goal But what is that actual end goal What is your framework trying to achieve
Rebecca Salganik : The setting that we're working with is called playlist continuation where the idea is that a user is starting to build out a playlist and we only know the information that they gave us about this playlist So maybe they gave us what we call seed tracks so maybe like one or two songs that they have put in this playlist And now what we want to do is to make suggestions for what else belongs in that playlist
Rebecca Salganik : But again because we don't necessarily have information about some of these songs or some or the user preferences we're we can't just use normal collaborative filtering So what we do is first we take our entire catalog of songs and we pass them through this so we pre-trained this model and now what it can do is it can embed songs So for all of the songs in our catalog we pass
Rebecca Salganik : Them through this pre-trained module and we have these representations And then we look at the seed tracks that the user has already given us and we try to find the K nearest songs in the embedding space So let's say we want to suggest 10 5 20 that's going to be K That's our hyperparameter And then what we want to do is we want to build a playlist for them which has like a set of suggested tracks
Kyle Polich : Right now the headlines are chock full of data breaches and regulatory rollbacks making us all vulnerable but you can do something about it
Kyle Polich : Delete Me is here to make it easy quick and safe to remove your personal data online Your data is a commodity Anyone on the web can buy your private details This can lead to identity theft phishing attempts or harassment But now you can protect your privacy That's why I've been using Delete Me One of the best things about the services when you first sign up they give you the flexibility to
Kyle Polich : Start with just basic information You choose what details you want them to protect Some removals happen within 24 hours while others might take a few weeks but DeleteM manages it all They keep you informed throughout the process and their quarterly reports show you exactly what they're doing to protect your privacy Take control of your data and keep your private life private by signing up for Delete Me
Kyle Polich : Now at a special discount for our listeners get 20% off your delete me plan when you go to joindeleteme.com/data and use the promo code data at checkout The only way you get the 20% off is to go to joindeleteme.com/data Enter code data at checkout That's joindeleteme.com/datacodedata
Kyle Polich : Much like any research effort you're gonna need some data set to work on to build your system But to the best of my knowledge the majority of these especially in music recommendation systems they're developed in industry That usually means they're proprietary they're private You may not have access to the thing you'd most like to have What sorts of data sets are you able to get access to
Rebecca Salganik : So when you work in academia and recommender systems one of the challenges is getting data sets and in music this is even more challenging because of copyright issues So what we do is we work with two different data sets that have been released publicly So in this project the data sets that we use are the Last FM data set and the Million playlist data set Both of them come from
Rebecca Salganik : Proprietary companies that essentially released their data publicly for the academic community to use So what we mostly focused on is the Million playlist data set which was released in 2018 by Spotify This data set is basically structured around playlists made by users And so the idea is that we want to perform what's called playlist continuation which is like a really important task
Rebecca Salganik : Now in music recommendation in general because people don't listen to albums anymore They listen to playlists which is basically like them creating their own album around some theme And so in this particular project we were just using the metadata that is associated with a playlist when we were talking about the textual descriptions that go with the audio
Rebecca Salganik : But in the future world we can see kind of the need for something more complex there
Rebecca Salganik : Right so when we think about the ways in which people connect with music they're often using what we call semantics So like when we professional musicians describe music they often describe the music in terms of the valence or the tempo or the BPM or the modes or something about the instruments And so when we were working in our experiments we had access to Spotify as like audio characteristics which talked about kind of the instrumentalness or the
Rebecca Salganik : Vocalness but what we can find from like human computer interaction studies is that when normal people talk about music they often talk about the atmosphere that it's evoking and that can be kind of like a semantic idea right For example this song makes me feel like I'm drinking a pumpkin spice latte or the context in which someone was listening to something So at the moment we have like very high level granular contexts right So like sports or something like that but
Rebecca Salganik : We don't have like I listened to the song when I quit my 9 to 5 corporate job or I listened to the song when I broke up with my toxic ex-girlfriend So these are the kind of like really granular semantic information that people use to connect with music And I think that the next frontier in music recommendation is being able to design algorithmic systems that are able to perform the kind of contrastive learning that I talked about in LARP but on like
Rebecca Salganik : More complex like textual descriptions of the audio So one of the extensions to this project that we're working on right now is where do we get this semantic information So in a perfect world I mean if you were in an industry setting you could have AB tests where you opened yourself up to this kind of explicit feedback from your listeners So listeners could like type in what they're looking for For example I'm looking for a song to pump
Rebecca Salganik : Pump me up to quit my horrible corporate 9 to 5 right Um and then the algorithmic system would be able to kind of perform the kind of contrastive learning we were talking about based on this really explicit feedback But since I'm not in the industry I'm in academia this doesn't exist So one of the projects that I'm currently working on and it's under revision is designing a data set that has this kind of semantic information So first what I did is we went through all
Rebecca Salganik : Of the different kinds of ways in which people talk about music and we presented a taxonomy of the different kinds of things that we want to see in future data sets what kinds of ways do people talk about music So there is obviously descriptive which is when people describe the music for example the bass in the track or the BPM of the song And then there are other semantic categories like atmospheric or contextual or situational So for example atmospheric would be like
Rebecca Salganik : This song feels like a warm hug right So it's kind of some adjectives that describe a feeling a vibe an aesthetic Then there's also contextual This song sounds like Dua Lipa mixed with Ariana Grande mixed with Bruno Mars Or you can have something like situational for example I listened to the song in the summer driving in the car with my windows down
Rebecca Salganik : And so once we have this taxonomy um what we do is we go on Reddit and we scrape Reddit all of the music threads where people talk about music and we try to find what song they were talking about and how they described it And so what you can see is that
Rebecca Salganik : There's like a whole huge amount of discourse uh organic discourse that people are kind of contributing where they talk about the ways in which they connect with music which is very different from the ways that like professional musicians would annotate a song And so this data set is actually already available on Hugging Face It's called Musicem
Rebecca Salganik : Music semantics Hopefully the paper that describes it and all of the experiments we ran on it I mean that's under revision but it will hopefully be out in the future And the idea is that this kind of data set can facilitate lots of different kinds of tasks because I mean as I mentioned before multimodal learning and contrastive learning between like text and audio is a really hot kind of direction for research in general but one of the things that's been
Rebecca Salganik : Lacking is access to this like really granular semantic organic text annotations And so let's say you wanted to use it for recommendation or other retrieval tasks that's definitely something you can use But also if you wanted to talk about like other frontiers right So there are lots of people who are saying that the future is like generative art that there's going to be a lot of like music generation and already there
Rebecca Salganik : There are many many models where you input text and it generates you audio um based on that text So when you talk about how you want that text to sound you also need a model that is like aware of these kinds of semantics and often underlying the generation process there is also this like contrastive learning where you're training the model to associate like some textual descriptions with some audio And so this data set is also really compatible with that
Rebecca Salganik : And then thinking even further down the line of like music videos or adding a third modality because the data set is compatible with like the unique identifiers that are used both on Spotify and in YouTube
Rebecca Salganik : What you can do is you can find like the music videos that are associated with the song and try to train a model for that So I think that is kind of the research direction that I'm hoping to work on and the way that that ties into discovery is that in order to be able to facilitate discovery as I've mentioned earlier like you need to be able to interact with the user and say what about this song did you like But when they give you the answer
Rebecca Salganik : You have to you have to have a model that understands that right So if somebody said I like this song because it made me feel something right Uh for example one of the entries of the data set that like really touched me was we listened to this song while my girlfriend had chemo So why do you like this song I like this song because it made me feel supported or close to her or happy or hopeful That is not an an audio description
Rebecca Salganik : Right That's not like because the BPM is 120 It's much more complex than that And so we need this kind of data in order to facilitate these kinds of interactions between users and algorithmic systems And hopefully if you can explain yourself better then the model can understand you better and then you can discover more music that has more I guess the kinds of recommendations it'll make will be more complex
Kyle Polich : Yeah it makes perfect sense that you want a lot more semantic data just knowing like the key the music is is I don't think anyone's partial to F major or something like that These easy signal processing features don't really tell the full story Going out to Reddit is a pretty clever idea but it also gets you into sort of an unstructured data problem What do you do with the crawl you've done
Rebecca Salganik : Yeah so that was basically one of the major challenges that we had to overcome right So how do we define these semantic categories and then how do we like meaningfully extract them So what we did is we did a bunch of ablation studies with lots of different LLMs and we tried we made our own synthetic data set and we looked to see which ones were able to extract the information we wanted So we tried to refine the prompts that we were giving them So first
Rebecca Salganik : We had to define these the taxonomy right So semantic situational atmospheric contextual descriptive what we mean by that And then we looked to see how well these models were able to extract that information from our synthetic data And so from going from that we chose the GPT 40 model We tried some reasoning models we tried cloud we tried some open source models and in the end we decided to go with GPT
Rebecca Salganik : And so we had GPT extract portions of the text So it's not rephrasing them It's giving instructions to just explicitly take the portions of the text that are associated with some semantic tag So the same block of text can have many kinds of semantic information For example this warm this song makes me feel like a warm hug because I listened to it while my girlfriend was going through chemo So that
Rebecca Salganik : It would be something like situational because it's a situation in which you listen to it but it would also be the atmosphere because that's how it made the person feel Um and it would extract those portions and then what we would do is we would also extract all of the songs that were mentioned in the post So maybe someone would say you know these four songs they would list like title artist title artist title artist make me feel blah blah blah And so we would extract all of them
Rebecca Salganik : Those pairs and then we would look for them on the Spotify platform So if you can't find a song that's a good indicator that your LLM has hallucinated so we would throw those out But if we can find the song on Spotify then you can get the unique Spotify identifier and then there is a database where you can map that to YouTube So if it exists both on Spotify and on YouTube then you've kind of got a pretty good proxy for like this is valid it picks it
Rebecca Salganik : Um and then we would scrape the audio and then that's kind of the entry in a data set So all of these semantic tags And then also if you want to like maybe reformulate these semantic tags that we've extracted into like one long sentence we would have another LLM that would do that And then we would use the other LLM to check the extraction to make sure that the the extraction was valid to some extent So of course whenever you're using
Rebecca Salganik : LLMs to do any form of like large scale extraction you have no guarantees that your data set has no hallucinations but we did our best and in the end the data set is about 35,000 entries and then we manually read over those entries
Rebecca Salganik : Because of course like publishing Reddit data is can be sensitive and this is something that we were really careful with So we wanted to make sure that there were no for example PII like identifying information that someone might have I mean music is not the most sensitive topic but it can get personal Um and so obviously Reddit is anonymized which is why we chose Reddit in particular to like have as much
Rebecca Salganik : Anonymity to these people as possible We also cut off all of our scrapes from like anything after 2022 is not in our data set because maybe that's too close to home maybe it's too personal and then yeah we just tried to make sure no one's name is anywhere no one's user ID and just to read over the entries to look for any form of hallucinations That was our scraping protocol
Kyle Polich : So it seems like MusicEm offers a pretty rich data set then Do you have a vision for maybe how other researchers might extend and build upon your work
Rebecca Salganik : Yeah so we published already the code for the pipeline for how we designed Musicem and we didn't scrape like exhaustively all the threads in music on Reddit in the world right We only actually took like 7 or 8 threads um because of course we have limitations in terms of our computational resources and there's a time horizon where we wanted to finish but we because we released this pipeline and all of the
Rebecca Salganik : Reproducibility information Hopefully people can continue to expand it Also um what I think is I guess what I feel most proud of is this taxonomy of the semantics right This understanding that just having descriptions is not enough that we need to be able to simulate like organic human discourse I think is really important particularly in the generative space
Rebecca Salganik : Where many of the current data sets that are available are very small in scale and also focus almost exclusively on descriptive information which is because they're annotated by professional annotators mostly musicians and the language between the way that musicians talk and the way that normal people talk forms this kind of interpretation gap Like the model just like won't understand the way normal people talk For example if
Rebecca Salganik : Somebody says I want a boopy base Like what does that mean Right Like you and I can we can picture what that means but the model probably won't be able to if it hasn't been exposed to that kind of adjective
Kyle Polich : If you had no constraints or infinite resources here where would you like to take it Uh there's 35,000 entries one dimension be add more entries or expand the taxonomy or who knows what Where would you take it next if you had more resources
Rebecca Salganik : So there's a recent paper that was published which has conversational multi-turn dialogues between a person and a system So the person says this is what I want and the system says is this what you mean
Rebecca Salganik : Uh but that is a very small data set and it's very I guess limited in it's semantic information So if you look at kind of the kind of information again that people are using there because the people were prompted to describe the music they're interested in using a specific template they mostly talk about like genres and artists I would really love to be able to get
Rebecca Salganik : Or maybe simulate from Muem a set of conversations between a person and an LLM where the LLM starts asking what I would consider to be insightful questions that are not just like metadatabased right Like what genres do you like but more like what are you feeling today I think if you look at what Spotify Gezer and Apple are doing
Rebecca Salganik : Right now and also Pandora I think that there is a thrust to try and describe people's tastes using more complex adjectives rather than just genres and that's what I would really like to do with Musicem So to design conversational models that are able to explicitly query the semantic associations they're making with the music they're listening to
Kyle Polich : And do you find that there is I guess I'm thinking about my own experience with music and maybe the experience of others I have a guitar player friend who loves Zappa and I think he likes songs that are novel from a musicology point of view sort of one way to put it Whereas maybe it's more common people just want an emotion or something to be evocative It's a very personal experience overall How do you develop a good set of features that can apply to as broad of a group of people as possible
Rebecca Salganik : Yeah so that's a
Rebecca Salganik : great question right How do we suit different levels of expertise in music is also a really important research question but how do you even assess a person's level of expertise if you cannot ask them about it There's like another really interesting extension to musicE is maybe we can't have the we don't know who these
Rebecca Salganik : Users are So we don't know their level of expertise but there are specific threads that are geared towards amateurs and geared towards professional musicians right So for example threads where producers give each other feedback that language is much more technical So teaching models as well to understand a person's level of expertise will be a great indicator for how to serve their listeners
Rebecca Salganik : Listening tastes because something that you'll find is like there's a really big difference between an expert perhaps your guitar friend who says I like rock music and some random person on the street who likes rock music What they mean can also be really different Like the idea between like generalists and specialists is also something I would really like to address in the future
Kyle Polich : Ideally maybe you could launch your approach on one of the existing platforms And I I hope maybe one of them would choose to adopt your work in some way and then you could get some learnings there In lieu of that how do you evaluate the degree to which your framework is working
Rebecca Salganik : So we have a training set and we have a test set So what we're trying to do is so the way that that works is we have a set of interactions maybe playlists and all of the songs that were in them or users and the songs that they were listening to when we started recommending
Rebecca Salganik : Right And so what we can do is we can mask out maybe the last 10 songs of a playlist or the last 20 songs of a playlist right Here we use our K parameter the threshold how many songs we want to recommend basically And so what we do is we mask them out and then we recommend the songs that we think should go in those places and then we look at the overlap between the songs that we recommended and the songs that like were in our test set
Kyle Polich : Well let's say you were hired by one of those companies and they wanted to put your ideas to work What sorts of evaluations could you do in that scenario
Rebecca Salganik : Also a great question So yes in the industry what often happens when you want to evaluate uh whether or not an algorithm is doing well is you define some sort of proxy for user satisfaction based on your playlist
Rebecca Salganik : And then when you make your recommendation what you can gather is like real user feedback as to whether or not they interacted with your recommendations Every company will have its own standards and it will depend very much on the experiments that they want to run but we can think about something like we recommended you a song did you add it to the play
Rebecca Salganik : List right Or we recommended you a song Did you listen to more than 30 seconds of it For sure if you listen to 15 seconds of it and went mm no thanks then that's definitely not the right answer right But what I think you're pointing out here is that there is certainly a bridge a gap
Rebecca Salganik : Between offline experiments which is kind of like what we performed here right without any real user feedback and online experiments where you're actually like making recommendations and getting information back from a user
Kyle Polich : I know you may not know exactly what industry is doing but do you have a sense where you can compare your work to traditional approaches
Rebecca Salganik : So in our particular work one of the questions we were asking is is multimodal representation even useful because some of the streaming platforms don't necessarily use it or at the time when we were publishing this paper it wasn't necessarily like obvious that it would be useful
Rebecca Salganik : So what we did is we compared with some of the state of the art models that were trained on huge amounts of proprietary data to just learn information based off of the audio not audio and text but just audio
Rebecca Salganik : This is part of a larger research direction that I'm particularly interested in which is the idea of like you know harnessing textual input from users to be able to really interpret what about the audio they're interested in So what our paper showed is that yes indeed like
Rebecca Salganik : Text is very like combining text and audio is very meaningful and what we're hoping to work on this year essentially is to expand like how that textual in like the depth of that textual information So at the moment what we're working with is just metadatabase textual information as you rightly pointed out right Like just like
Rebecca Salganik : The song Viva La Vida by Coldplay But you can also think about uh future worlds where a user actually describes the song in their own language and you use that and you align it with audio in order to be able to better understand their understanding of the audio Like what about the audio did they even like
Kyle Polich : Do you have any thoughts on the direction in general research is going and maybe how work like yours can push recommender systems forward
Rebecca Salganik : I think I touched on this a little bit when we were talking about what do I wish I could see in like recommenders in the future So one of the major focus of my research kind of the overarching umbrella term that I focus on is this idea of discovery So as our consumption patterns have moved towards platform
Rebecca Salganik : This is kind of called like platformization right We no longer discover music by going to a radio store and finding an album and just kind of trying it What we do is we go on a streaming platform where the algorithm is basically supposed to play the role of our friend that is supposed to like find us songs that are really cool
Rebecca Salganik : But the issue with that is that a lot of people feel very frustrated by this ceiling that they hit with recommender systems where they feel like it's gotten to a place where it's just recommending the same thing to them over and over again or it's just zeroing in on one of their tastes and not their more niche tastes and it's not exposing them to new things right So lots and lots of users are looking to discover
Rebecca Salganik : And our algorithmic systems are not geared towards discovery When we were talking about the metrics that you normally use to evaluate recommender systems what we're talking about here is accuracy We're looking at like what you liked in the past how well can we predict that But if you think about it intuitively the point of discovery is not to recommend something that someone already liked It's to find something new
Rebecca Salganik : In order to be able to do that we have to move recommender systems away from implicit feedback and to prioritize things like popularity bias diversity facilitating niche tastes because without that we are starting to see people who are leaving platforms people who are frustrated with the quality of these algorithms because the algorithms are geared towards serving mainstream you
Rebecca Salganik : I think initially when we started talking about my work we framed it as a fairness question and indeed fairness is a tool through which we can fix this problem but the overarching question I try to focus on and answer in many different ways is what can we do to start facilitating better discovery for users
Kyle Polich : I once saw a talk from an Amazon engineer who said I presume jokingly that his goal was that eventually you'd go to the Amazon homepage it would display a single product and you would immediately buy the product
Kyle Polich : And I guess the analogy in music would be every song you introduce to me is my next newest favorite song and they keep supplanting the previous favorite song but that obviously seems sort of ridiculous when put that way What's the ultimate goal for music recommender systems What does success look like
Rebecca Salganik : Before I answer that I just want to say I want to push back a little bit on like what we should be expecting from a recommender system in music So it's funny that you draw this connection between e-commerce and like the recommendation of creative content because to me these objectives are fundamentally different People in music are actually looking for diversity If every single song was the most amazing song
Rebecca Salganik : I just don't think that's even humanly possible The point is that when your friend recommends you stuff
Rebecca Salganik : Actually what happens often is that your music taste stretches to accommodate that Sometimes it takes 45 listens to really get a song right So the idea of diversity is part of the enjoyment of a song and the idea of being recommended something that you don't initially like is also part of that journey with creative content
Rebecca Salganik : Meanwhile if I'm shopping on Amazon and I'm looking for toilet paper I do not want the algorithmic system to say hey have you thought about plungers Because I know that I want toilet paper right And also if we think about the effort that a user has to put in in order to engage with the product on Amazon like they have to buy it
Rebecca Salganik : But in music like all you have to do is listen to 30 seconds of it So I think that the fact that e-commerce shapes so much of what recommender system literature kind of identifies as a gold standard is really hurting the recommendation community that's working on creative content because it's really really different
Kyle Polich : Well the success of any recommender system is going to be at least somewhat dependent upon input from the users whether it's implicit or explicit thumbs up thumbs down kind of stuff or maybe listen time What sorts of signals do you think would be most valuable
Rebecca Salganik : Yeah so you asked a question I went on a like a monologue and didn't answer the previous one where you're like how far away are we from an algorithmic system that's really able to like write like gold standard help us discover awesome music So I do think that we can achieve that and I do think that there is a model that is able to do that but that
Rebecca Salganik : model needs to be able to interact with us because it needs to be able to get that kind of feedback that you're talking about right Like this is what I want today this is what I'm looking for Even the idea that the same person can want different things in different genres like maybe like in pop I'm like yeah show me whatever but like for example in jazz I'm super knowledge
Rebecca Salganik : about jazz and I know what I like and I'm going to tell it to you So that's why we started with LAARP So LARP was the first step in this kind of like multimodal alignment for retrieval The next step is to have a model that can that can you say like I like the bass in this song
Rebecca Salganik : It can understand like as the sentiment bass as the like identifier for the acoustics It can then go into the audio for the song extract the bass find other songs that are similar to that bass serve it back to you and then be like OK did I get that right And have you say no I
Rebecca Salganik : I want the base to be more booby and then to have it start all over again So yeah I think we can have a model an underlying model that serves all of these needs right How we learn these embeddings doesn't necessarily need to drastically change but how we understand what someone is looking for there's a lot of room for improvement in that I think
Kyle Polich : Do you have a vision for how we bring that to the consumer
Rebecca Salganik : Yes yes absolutely I would like to build a user interface where users describe what they like about a song and then interact with the model and then tell you whether the model did it right or not In 2025
Rebecca Salganik : We just got like one small data set that looks like that I think it's called the CPDC It's the conversational collection of playlist continuation conversational dialogues about playlist continuations but it's very small
Rebecca Salganik : And when you train a machine learning model you need bigger ones So that's what I'm hoping for in the future but I think we're not there yet because as far as I understand um Spotify Deezer Pandora these are this is like happening right now These are the beta tests that I think are happening right now
Kyle Polich : Is there anywhere listeners can follow you and your work online
Rebecca Salganik : I have a website which is my full name Rebecca Salgani.com where I post my papers and sometimes my thoughts and then also I run a reading group on fairness and information retrieval It's kind of the topic that is started initially and now it's kind of just like fairness and representation
Rebecca Salganik : Multimodal representational methods and we have a page It's called Def D E F I R S T and we have a YouTube where we have like speakers come in and I think that's that's where you can find me
Kyle Polich : Rebecca thank you so much for taking the time to come on and share your work
Rebecca Salganik : Thank
Rebecca Salganik : you so much It was lovely to be here