Eye Tracking in Recommender Systems
In this episode, Santiago de Leon takes us deep into the world of eye tracking and its revolutionary applications in recommender systems. As a researcher at the Kempelin Institute and Brno University, Santiago explains the mechanics of eye tracking technology—how it captures gaze data and processes it into fixations and saccades to reveal user browsing patterns. He introduces the groundbreaking RecGaze dataset, the first eye tracking dataset specifically designed for recommender systems research, which opens new possibilities for understanding how users interact with carousel interfaces like Netflix. Through collaboration between psychologists and AI researchers, Santiago's work demonstrates how eye tracking can uncover insights about positional bias and user engagement that traditional click data misses. Beyond the technical aspects, Santiago addresses the ethical considerations surrounding eye tracking data, particularly concerning pupil data and privacy. He emphasizes the importance of questioning assumptions in recommender systems and shares practical advice for improving recommendation algorithms by understanding actual user behavior rather than relying solely on click patterns. Looking forward, Santiago discusses exciting future directions including simulating user behavior using eye tracking data, addressing the cold start problem, and translating these findings to e-commerce applications. This conversation challenges researchers and practitioners to think more deeply about de-biasing clicks and leveraging eye tracking as a powerful tool to enhance user experience in recommendation systems.
Guest
Santiago de Leon: I am a Spanish American doctoral researcher at Kempelen Institute of Intelligent Technologies in Bratislava, Slovakia and also part of the MSCA Eyes4ICU doctoral network (most prestigious EU grants completely funding a PhD). I have a master’s degree in Health Information Engineering from the University Carlos III of Madrid, with previous research experience in physics, neurobiology, chemistry, and mathematics. I began university level research in high school, as I was selected for a gifted and talented program. At my undergraduate institution, the University of Kentucky, I received the highest merit-based scholarship and graduated with three bachelor degrees in 4 years: mathematics, chemistry, and Spanish. In 2018, I joined two ongoing research groups in Madrid, Spain, one in psychiatry at the Jiménez Díaz Foundation and the other in machine learning at the University Carlos III of Madrid, where I focused on modeling psychiatric patients through their data and was a clinical research head. I have multiple publications spanning the fields of mathematical number theory, psychiatry, and machine learning. My current research focuses on user modeling within recommender systems, understanding how users browse and interact systems with eye tracking data to build interface-aware and gaze-informed recommenders.
Transcript
Kyle Polich : Welcome to Data Skeptic a podcast exploring the methods use cases and consequences of recommender systems
Kyle Polich : Welcome to another installment of Data skeptic recommender Systems You know recommender systems wouldn't be worth anything if there wasn't user feedback That's essentially the cold start problem and while there's some ways to mitigate the cold start problem those all seem to end up requiring again user feedback
Kyle Polich : Now your classic form of user feedback is a rating I always liked the stars I thought 5 things are more expressive
Kyle Polich : But I've been led to believe and I haven't checked this for myself but I've seen papers that seem to imply a simple thumbs up thumbs down system has just about the same information content as 5 stars
Kyle Polich : But it all comes down to a click The user has to select something for feedback to be gathered
Kyle Polich : Or do they
Kyle Polich : You know the iPhone is rolling out some new technology for eye tracking and it stands to reason all other handset manufacturers will follow suit Should the public want it our devices can now easily track our eye gaze
Kyle Polich : Or at least that's coming up on the horizon fast Luckily there's lots of researchers like the one we're gonna hear from today who've been looking into eye tracking technology Using it Santiago our guest today and his team have gathered data on exactly what people look at when browsing recommendation interfaces
Kyle Polich : We talk about that classic Netflix style layout where there's rows of seemingly related things and then if you're kind of done with that row you can go to the next Of course we see that format in lots of places so these insights probably generalize pretty well
Kyle Polich : Without further ado let's learn about eye tracking and recommender systems
Santiago De Leon Martinez : My name is Santiago De Leon Martinez and I'm a Spanish American uh researcher at the Kemplin Institute of Intelligent Technologies and also at Brno University of Technology and I'm also an uh MSCA uh doctoral student part of the Eyes for ICU doctoral network and I'm just about wrapping up my PhD
Kyle Polich : Can you share a few details on what the focus of your PhD is
Santiago De Leon Martinez : So my doctoral network that I'm part of is an eye tracking network mixing psychologists and computer scientists and my doctoral topic in general because it's funded by the EU through these prestigious doctoral networks it has to be somewhat related to eye tracking But originally when I started I wasn't as familiar with eye tracking but I'm an AI researcher and I focus on implementing and using eye tracking to build better informed recommender systems
Kyle Polich : It's not often I speak to someone that's in a close collaboration with psychologists from like a computer science AI background Can you talk a little bit about the collaboration
Santiago De Leon Martinez : Within the consortium itself I've published a paper for example with a group from the University of Ulm and they are a bunch of just pure research psychologists where we use pupil data to predict lying in a specific kind of psychological card game that they had where they have to bluff and do these different things I'm not a psychologist myself but I am a psychiatric researcher and I have many publications in psychology
Santiago De Leon Martinez : Psychiatry pure psychiatry as well as being an AI researcher and I continue doing this psychiatric research that I used to do when I first originally moved to Europe I did my master's in Spain at the University of Carlos III in Madrid and there I joined an AI research group in the university and then I was also a clinical researcher in psychiatry part of the
Santiago De Leon Martinez : He met at the Foundation University Hospital which is the largest private chain hospital within Madrid so we have a huge catchment area of patients And so I kind of really started out with this journey like mixing psychiatry and AI and trying to understand people through data but particularly psychiatric patients So I have quite a bit of experience with that
Kyle Polich : Well eye tracking is something uh one of the appeals of it is I think people intuitively get what you mean but I don't know that everyone has hands-on experience having their eyes tracked or looking at eye tracking data Can you share a few details about the mechanics What data do you capture
Santiago De Leon Martinez : Eye tracking when it gets down to what you capture is what we call gaze So gaze is the raw eye tracking data and like in the simplest terms possible what gaze is is an x y position on the screen that someone is looking at with a time stamp
Santiago De Leon Martinez : When it comes to this raw data which is called gaze we tend to process it in certain different ways because your eyes actually move like a ridiculously very very fast and eye trackers tend to be like something like 90 hertz or higher They can be 60 hertz maybe minimum but they tend to be 90 hertz or higher So the amount of data that you gather of this raw gaze data is just absurd Like for it to be like humanly intelligible in some way you have to process it in some way
Santiago De Leon Martinez : And we usually do that through fixations and seconds So fixation is what we classify as a set moment when someone is looking at a certain region within a certain time window
Santiago De Leon Martinez : That's effectively how you calculate There are many different ways within the eye tracking research to calculate fixations You can do it through velocity thresholds or different distance things but in general what it boils down to is like these are the areas in the eye tracking data There's this huge amount of sequential data that we've narrowed down to say like oh this for sure we know someone is looking at this point or this kind of small area
Santiago De Leon Martinez : What is classically done with fixation data is you filter it above 100 milliseconds so anything below 100 milliseconds you say it's not related to a cognitive process because it's just too fast for your brain to really process it
Santiago De Leon Martinez : So that's kind of like the basics of eye tracking data and what you do to kind of process and start to get an idea but even then when you have the fixations
Santiago De Leon Martinez : It's very hard to if you were to look at the data sheet it's still absurd It's not very humanly intelligible so we start to overlay it with pictures and do different types of processing so we can try to draw the narrative out and figure out how people in my case are interacting with a carousel or a Netflix-like interface
Kyle Polich : Well in the context of like an experiment like you're gonna put movie posters on the screen I have how you know N number of rectangles and it seems like you know maybe in that case you'd get well certainly cleaner data but how noisy does it remain I don't think
Kyle Polich : The human eye is as consistent as maybe uh we'd hoped to get really clean data Do you have clear indications that are sort of straightforward when you look and say like oh they looked at this movie poster for this amount of time or is there still a lot of the eyes darting out in random directions
Santiago De Leon Martinez : There there for sure is still some times where you have fixations on what we would say like background or non-AOIs so we refer to AOIs as areas of interest and those would be like the movie posters in Netflix like if we're talking about that Uh there are still fixations on the background or non-AOIs and in general we tend to just like kind of throw them out but you can still easily process the data and aggregate it to get like kind of these durations
Santiago De Leon Martinez : When you look at it like purely sequentially it probably it's very hard to understand like as a human That's why you do these aggregations on a lot of things or you overlay the eye tracking data over like the image of what the web page looks like and then actually the main thing that requires a lot of work is linking kind of the Xy interface position or you know Chrome page elements positions with the actual eye tracking data So
Santiago De Leon Martinez : Then you can automatically process the eye tracking data knowing this fixation was associated exactly with this movie poster because you have the Xy bounding box of the interface movie poster so that requires a lot of work It definitely can be done and it gives you a lot of very rich and powerful information to start to work with to figure out how people browse these types of interfaces
Kyle Polich : Well I obviously wanna focus on recommender systems as the use case for most of our conversation but before we jump right in could you share some details on what you've learned about in terms of other popular use cases of eye tracking outside of recommender systems what do people use this technology
Kyle Polich : for
Santiago De Leon Martinez : From the researchers that I know from my consortium there's people that work a lot on understanding eye tracking and autonomous drivers sorry not autonomous drivers but learning how people drive and professional drivers So in this case they have like a lot of experiments or even construction workers that are trying to do like a tractor or these different types of things or pilots and airplanes all these different types of professionals that learn how to
Santiago De Leon Martinez : Through years learn the expertise of how to figure out this machine And for example in an airplane you have a bunch of like dials and things that you have to constantly look at and there's a lot of eye tracking studies particularly that my my colleagues in psychology look at how does an expert look at all the dials and all the different things in an airplane versus someone that's a novice and effectively like experts start to learn tricks and shortcuts
Santiago De Leon Martinez : And like repeating patterns that they do to like look at these certain areas of the airplane to check that everything is going while the novice like kind of freaks out or maybe takes longer He doesn't have these patterns and all these different things So that's one example There's a lot of other researchers within the consortium work on
Santiago De Leon Martinez : Trying to understand saliency so how particular things can draw your attention in images and this even is done with a lot of computer vision and AI techniques where you have deep neural networks that learn saliency This is not something I've done with my research but it's a possible future direction as well of trying to understand how for example in a recommender system
Santiago De Leon Martinez : Whether positional bias of like where you place the item within the interface if that's actually stronger than the visual bias if we're particularly talking about like movie posters or even like on Amazon Marketplace like how good the picture is of the lawnmower
Santiago De Leon Martinez : And it's hard to say From my knowledge there is no paper that has looked at the comparison of like what is more important the positional bias or the visual bias We know both of these things are very important but there's no paper that really compares them and really says like oh this might actually be more important It's a very interesting research question
Kyle Polich : What are the applications for eye tracking in recommender systems
Santiago De Leon Martinez : So that's exactly what my focus is on I focus particularly on Carousel interfaces which are the Netflix-like interface and my main idea is to take eye tracking So on the first step that you can do is when you like eye track people that use recommender systems you can start to understand how they browse
Santiago De Leon Martinez : And so just based on the insights that you get from the eye tracking analysis you can start to get an idea of how people sequentially browse the different items and that can really inform your recommender system So one of the papers that I published within SIGIR in 2025 presents results like the first eye tracking study within carousel interfaces
Santiago De Leon Martinez : And here already it started to hint at a result that I found in a later paper that is being under review right now but effectively what the eye tracking analysis shows is that within this Netflix interface when you are first looking at the first page of items before swiping so like imagine you open up the home page of Netflix and you you can swipe each of the horizontal lists to see more items or not After swiping on the right side
Santiago De Leon Martinez : People don't start back on the left they stay on the right and they browse right left rather than left right
Santiago De Leon Martinez : So the normal traditional assumption within a recommender system how you position your ranked items on the first page it's 1 from starting from the left to right it's 12345 maybe 6 We say there's 6 items and then you swipe right and then it goes back to the far left and that would be the 7th 8th 9th and 10th 11th 12th However the eye tracking studies shows that people initially browse
Santiago De Leon Martinez : Starting on the right side So that actually gives a strong argument for considering at least you don't have to but it's it's worth considering flipping that ordering on all of the swiped interface
Santiago De Leon Martinez : pages
Kyle Polich : So are you saying that we don't read like typewriters we read more like snakes and starting from the right
Santiago De Leon Martinez : Yeah exactly
Santiago De Leon Martinez : Usually we would read left right like a typewriter over and over again but when it comes to these carousels that we swipe you keep your eyes like kind of close to where the mouse is and since you swipe to the right side this is on desktop interface you keep your eyes on that same right position and then you start from there and then you would go left
Kyle Polich : Interesting that's somewhat surprising Do you or maybe do you find it to be surprising and do you have an intuitive understanding of why it's like that
Santiago De Leon Martinez : I think it is surprising but it's not crazy It's not like out of this world right And I think the intuitive understanding is that you have this mouse cursor that draws people's attention
Santiago De Leon Martinez : And they're more likely since you purposefully move the mouse to the right side to go click the carousel to like swipe and see the next items that kind of draws your attention there and then you just kind of like let them slide in and you look at the same point that you were looking at before Yeah and it's probably the mouse or something like that that you're just like keeping your attention in the same place
Santiago De Leon Martinez : Because if you were to like kind of drag your eyes with it like it it moves pretty quickly like you don't see it fast enough Um so people start from that kind of same position
Kyle Polich : And this is sort of out of scope from your work uh but I'm just curious uh or maybe you could comment on who your uh subjects were in the experiment but in different languages I know they sometimes read right to left versus me as an English only reader I read left to right Did you have to account for that at all or were you just working with maybe English only speakers
Santiago De Leon Martinez : The study was done in Amsterdam and in Slovakia like European countries where people read from left to right So it's it's something we considered and absolutely like an eye tracking study of how people would behave in countries particularly in Asia where they read right left it would have to be very different right But at least for like left-right readers we think this this applies to them
Kyle Polich : Are there any other maybe uh nuggets of wisdom or efficient summaries you have from that about the way people browse Like my personal experiences I I think I kind of look at a row and then maybe I'll decide if I'm gonna skip to the next row or not and then scan like that Um I don't really do vertical scans myself but do you have any other insights along those lines of what typical people do
Santiago De Leon Martinez : Yeah there's a like I said there's a whole paper on this It's called Riding the Carousel It's on archive and I'm resubmitting it currently to a conference and I have a bunch of summaries there but the whole kind of summary of the results that I found is that people browse from the far right side So if you do a vertical transition so if you go from one list to the next one you're either going to do it from either of the ends the left or the right end So that's one thing So people transition very rarely from the middle movies
Santiago De Leon Martinez : So that's one thing So what I had mentioned about the swiping behavior afterwards that's probably the biggest result that considers switching the ranking order Uh when you look at the empirical like fixation data the amount of fixation like the amount of time someone is looking at a certain movie we actually get a different result If you wanted to base your rankings on that
Santiago De Leon Martinez : Uh Interestingly enough there is the 2nd to last movie on the right side slightly has is slightly fixated more than the last furthest right movie So that's another like kind of interesting result that kind of probably needs more eye tracking studies to follow up on and and really confirm this but that's at least what we saw through through our data set
Santiago De Leon Martinez : And the other results would be that we found there was a significant top two row bias so the initial kind of top two rows that are shown in the interface are definitely seen more and fixated more compared to the other rows
Santiago De Leon Martinez : This is kind of like an idea that for example if you have some sort of business constraint and you have some ad or something or you want to put your own materials maybe it's worth putting in the top two rows or just to get an idea These are probably the most important two carousels that you're going to have within the interface
Santiago De Leon Martinez : Another thing that we found is that like row skipping so completely skipping a particular row and not looking at any of the movies in it is very rare That's another thing as well So people may be kind of like looking at the topics and finding which topics that they like or not but they tend to at least look at a few movies and teach each of the topics
Santiago De Leon Martinez : And then finally we also have preference information from our users and to no one's surprise the topic preference which we gathered beforehand from a pre-survey is definitely very impactful on their browsing So people are more likely statistically significant way to browse towards areas that are in line with their preference metrics whether that's from rating or top their top genre or preferred versus non-preferred these metrics that we take
Santiago De Leon Martinez : That was seen across
Kyle Polich : that Well in a lot of the recommender systems applications that I've interviewed people about most of what they have to go on is just click through rate data so you display a bunch of items there's no information to them about what people dwelled on or looked at all you know is which one they happen to click on
Kyle Polich : Um which is is it's a good measurement you know it's it's it's what you've got and it clearly is some level of intentionality you're measuring from the user but it's much less rich than the data you have access to How do you leverage that information to make a better recommender system
Santiago De Leon Martinez : Yeah so one of the biggest things that you can even start to see from this data and how people browse is that people browse very differently and that gives you an idea that how you position items within the interface or what we call positional bias is super relevant Not only is it relevant it's particularly relevant when we're talking about complex interfaces like Carousels or like Netflix because you have like you have initial kind of
Santiago De Leon Martinez : Lists or carousels that are shown that you have to scroll that's a costly action to scroll vertically down to see more It's also somewhat of a costly action to swipe to see more movies within even within the same list
Santiago De Leon Martinez : And
Santiago De Leon Martinez : The idea here is that like all of these different positional like how you decide which movies you decide to show is already completely biasing the click that you're going to get right You could have you can find the most relevant for a certain query or the home page or whatever you can find the best movie for a user but if it's at the bottom and you have to like swipe a few times like the bottom carousel they might never see it right
Santiago De Leon Martinez : And this is also related to there's a lot of there's been also big research in impressions and impressions is not only like clicks but mixing the clicks with the idea that it was shown on the viewport or the interface
Santiago De Leon Martinez : To give kind of some sort of idea if it was seen or not Well eye tracking is like the next level It's even richer than impressions Like we know exactly which things were looked at on the screen and even if you want to start to play with the idea of duration or how someone is looking at it how many times they look at it to infer intent
Santiago De Leon Martinez : Or attraction you can start to do that as well I will say that when it comes to intent prediction or like attraction prediction and eye tracking outside of recommender systems there's some works in recommender systems but like outside of recommender systems in the eye tracking research world it's actually a very difficult problem and it's not so simple I personally don't work on that because how long someone looks at a certain item
Santiago De Leon Martinez : It can be up to so many different things and like contextually speaking and be up to the user in different ways You could stare at an item because a particular movie poster because I don't know it looks super awesome it looks like a very cool action movie or maybe it looks terrible like maybe it looks so bad that it draws your attention to it and you're like wow this movie is really bad it's garbage
Santiago De Leon Martinez : So how do you kind of distinguish between those two scenarios and it can be a lot of different things and it could be completely dependent on the user So I think that part is a very interesting research question but very hard very hard
Santiago De Leon Martinez : Although I do have one work of using gaze data for implicit feedback so using like how long people looked at it um in a collaborative filtering model um and at a particular item or movie and if you add those in as like positive feedback it helped the collaborative filtering model a bit with the recommendations It did improve the recommendations of the users but
Santiago De Leon Martinez : I think this is something that probably needs to be studied quite a bit more and it's
Santiago De Leon Martinez : Probably very difficult I think when it comes to the sense that if we have consumer grade eye tracking technology whether that's like the front facing iPhone which I don't know if you know from the iOS 18 you can be eye tracked with the front facing camera of the iPhone so it does work um so it can be gathered technically but if we get to the world where everyone is using some VR headsets then we will have access to eye tracking data always
Santiago De Leon Martinez : And in this case you can start to build neural networks that are taking the sequential data and start to learn these like rip out these patterns from a bunch of different users and I think then it starts to get feasible to really understand intention and attraction in some way or at least the neural network can start to figure it out a bit but as humans it's quite difficult It's like too much data
Kyle Polich : Well maybe this reflects my age and when I was in grad school but my first instinct is hey we have to featureize this data set Let's calculate dwell times or like the count of times they looked back at something or the order but what you're suggesting with the neural network is maybe throw it into the machine and let it figure it out for itself Uh what's your perspective on how to use the data
Santiago De Leon Martinez : So in terms of the data we have a data set released and it's it's publicly available for research if you'd like to use it and it's summarized in pretty much what you said So like there all of the fixations are aggregated within and are marked to certain AOIs or movie posters so you know like the the item idea how long it was gazed at or how how long it was fixated at the duration and you have like this summarized across all the screens
Santiago De Leon Martinez : For all the users as well as you have like some extra user information the item information uh the cursor movements are also included in this data set as well as like a selection explanation as to why people selected this movie um how familiar they were with the movie and then like why they selected it So it could be because of the poster because the details all these different things So this is all included in the in the data set and it's available for people to play with
Santiago De Leon Martinez : And part of the reason is because there is no eye tracking recommended data sets to date and I wanted to work on this and I realized OK there's nothing so I have to do this
Santiago De Leon Martinez : I'm going to have to do the user study and I would like in the future if anyone wanted to work on this like they also can do it as well and I generally believe that this we're tending to kind of this future where it's definitely possible that a recommender system has kind of like online eye tracking data
Santiago De Leon Martinez : But when it comes to like what you were saying in the in the sense of yeah you can get dwell times for things and I think there should be some human oversight like looking at like eye tracking data a bit and there'll probably be people you know that work on this a lot but
Santiago De Leon Martinez : I think it just starts to get very difficult and that's why I'm leaning more towards kind of like a deeper neural network approach to really trying to figure out these type of things It's just the data is like really really massive It's just like a very rich sequential data and when it comes to being able to understand it's just too much
Kyle Polich : DeleteM makes it easy quick and safe to remove your personal data online at a time when surveillance and data breaches are common enough to make everyone vulnerable Do you think your data is private Think again Data brokers make a profit off your data every single day Your private details are being bought and sold on the web
Kyle Polich : The New York Times wirecutter has named Delete me their top pick for data removal services and for good reason
Kyle Polich : Delete me's privacy experts do the hard work for you removing your information from hundreds of data broker websites and sending you personalized reports showing what they found where they found it and what they removed Have you been the victim of identity theft maybe harassment or doxing
Kyle Polich : If you haven't you probably know someone who has I certainly do and that's why I take my online privacy seriously Using deleteM has helped me minimize the amount of personal information available to strangers online
Kyle Polich : It's one of the smartest steps I've taken to help protect myself in this digital age Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners Get 20% off of your Delete Me plan when you go to joindeleteme.com/data and use the promo code data at checkout The only way to get the 20% off is to go to joindeleteme.com/data
Kyle Polich : And enter code data at the checkout that's join deleteme.com/datacodedata
Kyle Polich : Thanks to this week's sponsor Zapier These days it feels like everyone is talking about AI including us here at Data Skeptic But if there's one thing you've learned from this show it's that talking about a new technology isn't the same thing as implementing it
Kyle Polich : And honestly AI has already changed how I work I haven't written a line of code in at least 3 months I just speak it into existence now It's not just about speed it's about what becomes possible when you can offload the tedious stuff and focus on the creative problem solving But here's the thing the real power comes when you can integrate AI into your actual workflows not just use it in isolation
Kyle Polich : That's where Zapier comes in Zapier is where tech innovators break the hype cycle and put AI to work for real I've been using Zapier for at least a decade and it just keeps getting better What I love most is how easy it is to incorporate into your workflow
Kyle Polich : You don't need to be a developer or to wait for IT approval We use it for podcast production handling marketing campaigns It's popular for sales pipelines and HR processes Zapier fits right in We're building automations that can handle everything from moving audio files between cloud storage to managing our content calendar and routing new business inquiries to the right team member So what is Zapier exactly
Kyle Polich : It's an AI orchestration platform that lets you bring the power of AI to any workflow You can connect top AI models like chat GPT and cloud to the tools your team is already using So you can add AI exactly where it needs to be Teams have already automated
Kyle Polich : Over 300 million AI tasks using Zapier Are you ready to join them Get started for free by visiting Zapier.com/dataskeptic I'm going to spell it for you That's Zapier Z A P I E R.com/data skeptic
Kyle Polich : Well I don't know that we named it but I think we were discussing the Rec Gaze data set which is really pioneering in this area as you said I don't know of any other so that's a good motivation to release it For other researchers who might be in this space and are thinking about using the data set can you describe a little bit about its contents
Santiago De Leon Martinez : So the Rec Aze data set is the first eye tracking data set within recommender systems and it's the 3rd feedback data set click feedback data set available for Carousel interfaces at least publicly available and I would also say it's like kind of the 1st
Santiago De Leon Martinez : Super extensive recommender data set because it also includes like all the cursor movements so you know when you have the time stamps of when the cursor was on something as well as when their eyes were on them and also when the click events happened So all of this you have like all this time stamped information as well as including like user features like age gender the genre preferences that we get from our pre-survey beforehand some other questions on expertise of how often they use interfaces like Netflix
Santiago De Leon Martinez : Um and then item features as well you can get like all the recover all the descriptions of the movies
Santiago De Leon Martinez : As well as IMDb scores if you'd like
Santiago De Leon Martinez : Um but the main thing is kind of like this gaze data plus the cursors plus the clicks that is just a very complete data set and then if you're interested in as well like why people pick this particular movie
Kyle Polich : Well I think what you're describing then would be sort of a a hybrid system that would tie in all of these features like I don't know that eye tracking alone can build a recommender system but eye tracking plus collaborative filtering is probably better than just collaborative filtering alone Curious to hear your opinion on it and maybe how you integrate these systems
Santiago De Leon Martinez : Yeah so like what we were talking about before when it comes to like actually using the eye tracking data as feedback and plugging it into a model I think that's like kind of the step in the future of it that we kind of need neural networks for that
Santiago De Leon Martinez : I try to do something simpler first right And the idea is the first step is like we have this eye tracking data we process it and we try to get human insights as a researcher looking at the data and how we should design the interface itself and maybe where we should place your items So the first kind of result that we got from that what I mentioned before is that maybe we should consider ranking differently after swiping these carousels So that's the first thing It's like human insights
Santiago De Leon Martinez : That you can then program as an engineer as a recommender systems engineer into your system that makes more sense And I think one of the big problems when it comes to research as well in practice for recommender systems
Santiago De Leon Martinez : Recommender systems are like very complex and we make like a lot of assumptions
Santiago De Leon Martinez : Here and there that actually add up quite a bit and as things have started to get more complicated it's more like more and more likely that these assumptions are wrong So back in the day we only had like search engine results pages and people assumed that people browsed top down Well that's actually a fair assumption because it's it's pretty much how the page is set up and there's not much else to do but you know we still did eye tracking studies and we you know we confirmed it empirically which is important for the scientific process
Santiago De Leon Martinez : And things started progressing Then there's like the Google Image search results that starts to get a little bit more complicated but maybe you assume left right However there are eye tracking studies that show that there's kind of like this F pattern behavior on these grids meaning that people are more likely to hit the left side and go further right when they're in the top so like in an F right and as they go further and further down they're less likely to branch off in this F right to go right
Santiago De Leon Martinez : There's another eye tracking study with particularly with images that shows that there's like a center bias as well so that's another possibility for kind of grid setups
Santiago De Leon Martinez : And usually you'd make the assumption that like you read left right right But it's not it's not too crazy or anything and but it could be a flawed assumption that effectively like takes a huge hit on performance on your recommender system Like you have the greatest recommender system possible that gives perfect relevance but you put the items in the wrong place you messed it up right It doesn't matter how good your algorithm is if you're putting the items in the wrong place
Santiago De Leon Martinez : So that's why it's kind of nice to do these empirical studies particularly by tracking to really understand how people browse
Kyle Polich : Well I know most movie posters they're similar they're all trying to do the same thing let you know what's in the film a little bit There's variety but not insane variety If there was I could see where maybe a movie poster that's all text maybe my eyes would dwell there longer because I'm reading
Kyle Polich : It or something like that and I'm sure at least one movie poster in the world exists like that even if it's not common but do you have concerns like that that there could be something almost spammy about a movie poster that would draw more gaze than perhaps the is a true conveyance of the user's interest
Santiago De Leon Martinez : Yeah for sure that's like definitely a big limitation to like our study and even the data set and it's something that we recognize Um most our eye tracking results are done through aggregate across like every participant does multiple screens They do 30 screens of free browsing
Santiago De Leon Martinez : And we aggregate all the eye tracking results across them so it's definitely possible that visual bias over a particular poster being either like super attractive for some reason that it's really calling to the particular participant or kind of what you mentioned in the sense that maybe if it was like all text it's maybe not the most attractive movie poster I would say like from the get-go but being kind of the ugly duckling maybe it draws your attention to it right Could be yeah
Kyle Polich : yeah
Kyle Polich : And could you describe the experimental setup a little bit if I had been one of your participants what would I have gone through in the data collection process
Santiago De Leon Martinez : Yeah so you would have either been in Amsterdam or in Bratislava in Slovakia uh at our institutes and you would have came in we would give you kind of like a pre-survey at the beginning This is where we get all your genre preference information your age gender these type of things as well like the user information
Santiago De Leon Martinez : And you kind of sit down in front of a laptop with an eye tracker and an eye tracker is like these little bar eye trackers that you put at the bottom of the screen and it effectively just like looks at you and will gather kind of these x y positions linked to the computer screen and you just kind of sit there and do an environment that we made that's as close as possible that we could get to Netflix you know without infringing on copyright And the idea is that like Netflix is like
Santiago De Leon Martinez : The most well known standard kind of like movie interface and if you look at like Prime Video or Hulu or something they're all kind of carousel interface and they're all pretty much like Netflix So we just opted for Netflix so you just kind of sit there most people take like about an hour to go through a bunch of screens where they're just picking a movie that they would like to see Amongst like all the ones available there's 150 available on each screen
Santiago De Leon Martinez : And they pick a movie they like to see That's the free browsing task We have some additional tasks as well to be able to relate how for example a user might based off of a different task they might search differently And there's a direct search task For example this is like we tell them to find a particular movie in a particular genre so that gives you an idea to be able to compare how these behaviors may be slightly different
Santiago De Leon Martinez : And finally there is a 3rd task It's called the semi-free browsing is what we call it and it's find a movie that you would like to watch within your favorite genre So it's this free browsing but limited to your favorite genre which you marked on the pre-survey
Kyle Polich : I don't know that I have shared this opinion but people have strong privacy concerns Uh I don't think most people would want the camera to come on while they're browsing their Netflix feed but maybe there's an argument for it if that can improve my overall Netflix or YouTube experience or whatever it is maybe I would want to opt in What are your feelings on the privacy aspects of this
Santiago De Leon Martinez : Yeah that's a that's a really good question Um I think ethically and like privacy wise it starts to get very complicated because we don't kind of consciously control our eyes
Santiago De Leon Martinez : When when we get to the point where people are like eye tracked constantly you can start to figure things out about them that they're not even like that conscious about either and it reveals so much information about you like semi-consciously or even consciously that you may not even know about that it's it can be pretty crazy There's other things as well like with eye tracking data some people are working on trying to diagnose Alzheimer's all these different things So like based on your eye tracking data from Netflix
Santiago De Leon Martinez : If it were to get released or something someone can try to diagnose you with that with some sort of mental illness or something like that right And I don't think people are happy with that
Kyle Polich : I feel like I at the right age I would want to opt in and have my phone warn me hey speak to a doctor you're showing
Santiago De Leon Martinez : That's fair that's fair right with consent you're like you knew what you're getting into right Like you knew what you're getting into but yeah if you're just normally using Netflix and uh Netflix is figuring out like every single thing that you exactly like more than like you even know that almost sounds kind of creepy um right
Kyle Polich : Yeah I mean it's it wouldn't surprise me to find out that a future study determined that through eye gaze data you could determine someone's romantic preferences um you know maybe that's science fiction but it seems quite plausible We could also take it a little bit sci-fi I don't think it can like you'd mentioned earlier maybe it can detect lying although that's uh uh I know traditional lie detector machines are are not science maybe the eye tracking lie detector is what are the limits here
Santiago De Leon Martinez : I I think to be honest we don't know the limits Eye tracking research is it's been done for a few years now but the technology keeps getting better and better and we haven't been to the point where we have like an online system that is constantly eye tracking where you can start to like I said plug in some sort of very sophisticated model to start to figure things out like really figure things out But like you said when it comes to attraction or sexuality
Santiago De Leon Martinez : It's possible that like your eyes can really reveal this based on these posters of like seeing you know whatever gender you like or attractive people these type of things
Santiago De Leon Martinez : It's definitely possible that the eye tracking data could completely reveal that And do you think I don't think most people would like Netflix or like kind of these companies
Santiago De Leon Martinez : Without knowingly telling them your kind of your personal information right like for that to be out there particularly when it comes to their sexuality or these different things I think that's very private and personal so I think people would probably be kind of scared about that But there also comes a sense like like you said that if these things integrate very easily into your life and it makes the system so much better
Santiago De Leon Martinez : It may be possible that this kind of practicality of it kind of overweights it and that's the same thing when it comes to cell phones for example like previously in my masters with psychiatric my psychiatric work with ML stuff we did data scraping of psychiatric
Santiago De Leon Martinez : Patients from their phones to get their step data their sleep data everything and we do this to do change point detection to see if they are getting better or worse during a certain treatment and possibly even diagnose some sort of mental illness or problem that they have But effectively data is like a biomarker for someone Someone can be represented through the countless amount of data that they generate throughout the day with their cell phone or all these different things
Santiago De Leon Martinez : And I think once you start like really getting access to a bunch of that you can start to like nail people down and identify them and figure out things about them that they probably could be kind of you know worried about right
Kyle Polich : Again out of scope but just curious to get your thoughts is my own eye tracking data unique like a fingerprint
Santiago De Leon Martinez : Uh it is it is yeah interesting Particularly the most dangerous thing when it comes to eye tracking data is your pupil data Uh the the data set that I have it doesn't have any pupil data on purpose because it's publicly released and it's so anonymized in a way to protect the identity and there's like consent and everything from all the participants They know you know that this is going to be made into a data set but pupil data is very dangerous in the sense those micro movements that your pupil makes can identify you
Santiago De Leon Martinez : Just like for example there's also research in very high frequency accurate data about your heartbeat can also identify
Santiago De Leon Martinez : you
Kyle Polich : Fascinating
Santiago De Leon Martinez : Yeah everyone's heart in general kind of works the same but there are slight differences and maybe the volumes and all these different things you can figure things out with a very sophisticated model yeah
Kyle Polich : Well I was gonna ask you uh what you'd recommend if Netflix hired you I'm gonna switch that to YouTube because in my personal experience they have the most room for improvement on their video uh recommendations Um if they were to take your findings seriously and brought you on board what would be the key advice you would give them
Santiago De Leon Martinez : Like I've mentioned before the most important thing that my research is kind of focused on outside of the eye tracking is like let's start to question the assumptions that we make when we're designing a recommender system like the basic assumptions that we make about a user Let's question those and like let's try to figure out if they actually make sense and how we position the items from the get-go That's one thing And really what my research is transitioning now is actually simulating users
Santiago De Leon Martinez : So eye tracking and clicks and all these different things you can build user models or click models these different things to try to represent a user and the simulation of users when you just have clicks it's kind of hard to really understand someone so you need to like get the information whether their preference these different things the different context but eye tracking as well starts to give you like a rich amount of data to be able to simulate a user and create more realistic users
Santiago De Leon Martinez : The problem is is like nowadays in the digital age that we live in
Santiago De Leon Martinez : We have so much data
Santiago De Leon Martinez : That's not the problem I think previously like maybe 20 years ago we were like we need more data we need more data Now we have an absurd amount of data Now we need to get good data and we need to figure out how to filtrate or you know what they're doing with all these tricks with LMs to get them to work better and better It's about getting the right data
Santiago De Leon Martinez : And when it comes to recommender systems now I think it's about making the correct simple assumptions from the beginning and then also trying to get good data and one of the possible ways is to simulate users correctly
Santiago De Leon Martinez : And this can be done through like particularly when you get multimodal rich data that represents a user whether it's from like the cursors as well as the clicks as well as the eye movements and their preference information all these different things So that's kind of the direction I'm going towards now
Kyle Polich : Could you share a few details about that simulation process at its rudimentary level I guess you could make random XY's but that's probably not helpful How do you model the richness of a true person's gaze changes
Santiago De Leon Martinez : Yeah so modeling like actual gaze data this is like a very recent topic Well it's been worked on for many years but only recently have researchers really been able to correctly regenerate like stimulate gaze data and this is from a paper from the University of
Santiago De Leon Martinez : Andreas Bolling's research group they recently were able to regenerate I think gaze data for biometrics for what we mentioned for like security like you know being able to tell a certain person how they look at something and identify them based on that I think it's the first time that they had a deep learning method that successfully was able to kind of like trick it
Santiago De Leon Martinez : And actually re-simulating the gaze data that's like a very very hard problem I work particularly more on kind of re-simulating the click which is simpler Yeah I think it's it's very interesting I think how you would try to rekind of build like how someone observes the page But as you've mentioned before one of the problems that like people like look in weird places you don't even know why and how do you simulate this Sure you can do like
Santiago De Leon Martinez : Add noise you can do a Bayesian you know some sort of probabilistic process of like they may do something weird something like this but it's very very hard to model I think
Kyle Polich : Well the classic problem in recommender systems is the cold start problem that a user shows up you've got no information how do you possibly make a good recommendation even once you get your first click what does one click tell you it takes a couple of clicks at least but it occurs to me the eye tracking data could be very beneficial to that cold start problem would you agree
Santiago De Leon Martinez : Uh so like a cold start user you have no eye tracking data of them correct
Kyle Polich : You know at 2nd 0 T equals time is 0 you have no data but with eye tracking you start acquiring data a lot more quickly than you would with just click data
Santiago De Leon Martinez : Yeah yeah so in that case yes if you're talking about like even before the click there's like how someone observes already that is like a rich amount of data that's when we're starting to talk about like maybe a recommender system that I said that's a deep approach that would start to look in an online way So you're browsing Netflix and one way so the carousels or these lists are pre-generated by Netflix and from my understanding every company pregenerates it for that particular user in that context but
Santiago De Leon Martinez : Once you have an online like eye tracker and you have this eye tracking data just flowing into the model
Santiago De Leon Martinez : Why don't you generate the recommendations based off of how long they're looking at a certain movie poster It makes perfect sense right So in that type of scenario I completely agree with you When it comes to like the true cold start you don't even have any eye tracking data on that user or anything like true time zero Probably the eye tracking data just allows
Santiago De Leon Martinez : You to get richer models of the other users where you get an idea that the collaborative filtering works more effectively because you've built a better user model of the other users you are able to maybe try to relate it to these richer models that you have of the other users and you'll probably get a better fit that way
Kyle Polich : One of the papers we've been talking about focus on movie recommendations in this carousel style but it's not the only place that style can exist it could be an e-commerce or things like that too Do you think your results will translate there seamlessly
Santiago De Leon Martinez : That's an excellent question Yeah like there's a probably a big difference between the context and kind of the intent of like being on an e-commerce site and a movie site
Santiago De Leon Martinez : I wouldn't be surprised at this idea of how like someone browses and after you swipe and you start left right and then you swipe and then you go right left I wouldn't be surprised if that's kind of on a desktop interface with a mouse I should say if that is consistent across all different types of domains
Santiago De Leon Martinez : And different types of interfaces where you can have like 10 carousels and like you can swipe 5 times or 3 times I wouldn't be surprised if that's consistent There need to be more studies really to follow up and to know for sure I I would guess that that's switching from left right to right left after swipe that's consistent but how people browse when it comes to like movies versus this kind of e-commerce I think it starts to change quite a bit
Santiago De Leon Martinez : Because like exploration for like entertainment purposes is very different than exploration for like a very concrete thing Like you open up Amazon because you want to buy a lawnmower you're not doing it for fun You're probably your exploration is limited in some way because it's like very task driven very specific You don't want to waste that much time like looking for a lawnmower right Well
Kyle Polich : you and I shop the same but not everyone's like that yeah
Santiago De Leon Martinez : Yeah that's true that's true
Kyle Polich : How important is the de-biasing of clicks in recommender systems
Santiago De Leon Martinez : So I haven't worked in industry yet I I do have an industry partner that I work with a bit um but I'm not super super experienced I've talked to a few people My understanding is that it definitely helps the metrics like doing some sort of positional debiasing so upweighting and down weighting based off of like where that item was positioned I'm not sure like how impactful
Santiago De Leon Martinez : I know it's most helpful for offline evaluation when you're trying to like you know we were talking about almost simulating what someone would do with a different policy or with contextual bandits all these different things Doing some sort of debiasing is huge when you're particularly doing off policy evaluation
Santiago De Leon Martinez : Um in this case it's like very important or you get like crazy results Uh it helps a lot with that When it comes to the actual data I believe people in the industry do like softer kind of positional devising things From my understanding I think it's most helpful when it comes to offline evaluation
Kyle Polich : Well we've touched on this a little bit but I definitely wanna ask what's the future for you Where is your work headed
Santiago De Leon Martinez : So right now I'm wrapping up my PhD and I'm looking at different kind of like internship positions possibly maybe at Netflix and because my you know topic is kind of in line with that
Santiago De Leon Martinez : And I'm looking to work in industry research so I've been an academic researcher since I was 15 years old I started in the University of Kentucky doing research and I've done a lot of research from chemistry physics neurobiology
Santiago De Leon Martinez : And I've been like a researcher my whole life so I think I would I still love research and I want to continue doing it but I think I want to try like a different atmosphere and a different environment than academia to kind of see what I like and figure it out and also help me grow as a researcher
Kyle Polich : And what's the future for the rec gaze data set
Santiago De Leon Martinez : I hope that the rec gaze datasets will be kind of the first data set that people go to that are really interested in trying to really understand how people browse maybe simulating users or as this eye tracking these eye trackers get better and better and the companies start to get like a bunch of gaze data that you know everyday users are getting eye tracked
Santiago De Leon Martinez : The problem is like these companies are probably not going to want to share this data with researchers like academic researchers and they'll probably be incentivized not to It's like a very private thing So but it will be a struggle for like researchers to be able to play with this and try to figure things out So if you're interested in what I call gaze-based recommender systems these are recommender systems that use the actual gaze as like an input to the model as a feedback
Santiago De Leon Martinez : Then you should definitely look at the Recgaze data set or if you're interested in gaze informed recommender systems this is instead of like what I was mentioning of building user simulators user models or using the insights that you find from the eye tracking data to build a better recommender system yourself then that's also a great approach
Kyle Polich : And is there anywhere listeners can follow you and your work online
Santiago De Leon Martinez : Yeah so you can find me on Research Gate and LinkedIn Those are the two main places and uh yeah also on Google Scholar you can always follow my
Kyle Polich : works For sure we'll have some links in the show notes for all of the above Santiago thank you so much for taking the time to come on and share your work
Santiago De Leon Martinez : Yeah thank you so much Carl I really appreciate it It was a it was a nice chat with you Definitely