Networks and Recommender Systems

Transcript

Kyle PolichAsaf Shapira

Kyle Polich00:05You're listening to Data Skeptic Graphs and Networks the podcast exploring how the graph data structure has an impact in science industry and elsewhere

Kyle Polich00:17Welcome to Data Skeptic and sorry for the delay It's been a while since I dropped an episode and that's unusual for this show We're gonna get back on our weekly schedule shortly The fact of the matter is I have just been swamped but I went on a hiring spree We have a new editor new production coordinator new director of technology so I'll be able to pay attention to the podcast and

Kyle Polich00:39Put out the show you guys deserve shortly We've actually been planning to ramp up the next season of Data Skeptic partially to blame with some scheduling conflicts I got one interview that's coming up in a few months that is such a good one we'll just do it out of sequence You'll hear the graphs and networks music again one day But no the new season is going to be all about recommender systems So this episode is me picking a soft brain about how a network scientist looks at this problem In fact I've already done a bunch of interviews

Kyle Polich01:09I just had a few ducks to get in a row before we could kick this off so I'm very excited for this season I should probably actually plan a little break in between seasons I always feel like these end with no pomp and circumstance It's like on Doctor Who when Adri died at the end and then they don't acknowledge it So at least I rope us off into this little let's say call it crossover or prelude to recommender Systems episode

Kyle Polich01:34Straddling both topics

Kyle Polich01:37And there's tons more topics as you can imagine not just the methodologies of how do recommender systems work but frontier stuff like federated recommender systems and and privacy centric systems and things along those lines tons of interesting stuff as you will hear when I get into the segment with Asaf He astutely points out that a lot of the cutting edge stuff is going on in industry And don't get me wrong there's still plenty of researchers out there that I found to interview That's primarily what we do here on Data Skeptic

Kyle Polich02:06If you're actively working in recommender systems shoot us an email but there's some conditions First of all no CEOs no C-level anybody I only want to talk to the person getting their hands dirty the the data scientist engineer whatever your title is the boots on the ground person getting the work done

Kyle Polich02:24If you're open to a no strings attached conversation feel free to hit me up I am most sincerely looking for some industry use cases especially novel ones So if you're working in the field and wouldn't mind sharing your work here on this platform please email me at a special email we've created RS as in recommender system RS at data skeptic.com So thank you all for your patience as we've had a spotty schedule and thanks to Vanessa the most who's been a real hero in coordinating everything

Kyle Polich02:54Handling advertisers and whatnot We're getting back on track and I can't wait to share with you all the great work I've already heard done for our new season on recommender systems I guess starting right now

Kyle Polich03:07Welcome to another installment of Data Skeptic Graphs and Networks I'm joined as always with co-host Asaf and Asaf I've got some uh an announcement to make today This will be the first time the general public hears about this I think you're about the 3rd person to know but the topic for the next season is going to be recommender systems

Asaf Shapira03:26Why am I only the 3rd to know Oh

Kyle Polich03:28well I told my assistant and I told someone close to me and uh you're #3 I should have told the graphic designer first if I was being smart

Kyle Polich03:37And you know it's funny I asked Chat GPT to predict the next topic for data skeptic what our next season would be and uh it had some really good ideas actually one of which was recommender systems So I guess maybe it's overdue Wow cool OK And then of course so why are we talking about it I thought it would be interesting for you and I to explore the possible overlaps between network science and

Kyle Polich04:02Recommend your systems pick your brain a little bit on what the opportunities are there and see if you had any suggestions on research directions we should take along the way as we get deeper into that literature and start to highlight certain authors who are doing interesting stuff in that space

Asaf Shapira04:16Cool so you know

Asaf Shapira04:17Uh first of all the the applications for recommendation systems usually in in industry so there's lots of papers about it but I guess the cutting edge technology is out there You think so Yeah I guess right I'm from the academy I don't want to say uh uh to say how things I guess most of the money

Asaf Shapira04:40It isn't there right It's not there If I understand the current political wave it's not going to get better So I guess most of the uh you know resources and so on are in the industry So practitioners bring some interesting insights you know from people who actually work with this stuff

Asaf Shapira05:04So I'm a practitioner so I I really advocate it That's you know when we talk about who to interview but you know it's actually funny you said it because I just hung up on another Zoom call with my students who actually are doing a project now about recommendation system serendipity When you do a recommendation system

Asaf Shapira05:27You connect things that aren't connected right What are the edges there I like to categorize it into two parts like explicit connections edges and implicit So when I look at explicit I look at edges that there is no argue that that's a network When we talk right we are creating a network What defines these edges like physicists would say we exchange information

Asaf Shapira05:57Information is an edge or maybe a dependency could be an edge and so on things that are absolutely dependent on each other On the other side on the other side of the spectrum there's the implicit connections or edges That's where recommendation systems kick in because I like the let's say Marvel movies OK I like Spider-Man and I like Ant-Man I can assume that I would also like Captain America

Asaf Shapira06:26Although I haven't seen it so it's uh what we call a recommendation or a link prediction Yeah

Kyle Polich06:33it's

Kyle Polich06:33a link prediction in your mind yeah

Asaf Shapira06:35That's the kinds of edges that recommendation system work with right Implicit edges

Asaf Shapira06:41Since they're implicit you know it's not bulletproof right It's it's not perfect There'll be some some uh mistakes and uh what we tried with this project is try to minimize those mistakes and try to do all kinds of cook the networks where you can call it Just to give an example one of the most basic ways to do a recommendation system is using uh what we call a bipartite network

Kyle Polich07:09Building multi-agent software is hard Agent to agent and agent to tool communication is still the wild west It's clearly the emerging future but how do you achieve accuracy and consistency in nondeterministic agentic apps That's where agency comes in They have a very clever spelling Here's how it goes A G N T C Y I'll give it to you again in a minute The

Kyle Polich07:36Agency is an open source collective building the internet of agents The Internet of agents is a collaboration layer where AI agents can communicate discover each other and work across frameworks For developers this means standardizing agent discovery tools seamless protocols for interagent communication and modular components

Kyle Polich07:56To compose and scale multi-agent workflows build with other engineers who care about high quality multi-agent software See where you can fit in this ecosystem Visit agency.org and add your support That's AG NTC Y.org

Kyle Polich08:18When I talk about online privacy security I speak from experience as a delete me subscriber What's been eye-opening is watching my quarterly privacy reports show me just how many data brokers are out there collecting our information In my first report alone I discovered hundreds of listings containing my personal details Everything from old addresses to family connections I'm a low rent public figure but I don't

Kyle Polich08:42want the rest of that stuff out there What makes Delete me different is their comprehensive approach They don't just remove your information once they keep monitoring and removing it continuously and they stay ahead of privacy threats As new data collection methods emerge their team adapts and evolves their protection strategies They don't just focus on known data brokers they're constantly scanning for new sources of data exposure and developing

Kyle Polich09:05removal processes to address them It's like having a dedicated privacy guardian working around the clock Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners Today get 20% off your Delete Me plan by texting data to 64,000 That's data to 64,000 Message and data rates may apply

Asaf Shapira09:34To give an example one of the most basic ways to do a recommendation system is using what we call a bipartite network

Asaf Shapira09:42A network with two kinds of nodes nodes kind A and kind B Let's call it um in the case of a recommendation system of products let's call it nodes from Type A customers and from node type B products In this kind of network the only connections are between nodes A and B Customers aren't connected within themselves and so are products The only connection is

Asaf Shapira10:10Customer buying a product in order to understand which product might go with another product what we need to do is to project the network that is to delete the customers and their edges and connect the products by the customers that are linking them OK and then we get a one dimensional network a network of only products and the edges between the products are assumption We assume that if people bought watches they also bought

Asaf Shapira10:37Necklaces because we see an edge between them Let's say I bought a watch and I also bought cucumbers on the same buying rampage OK so there's a link between a watch and cucumbers but it was only me OK It was a random

Asaf Shapira10:56decision

Kyle Polich10:57Well earlier when we were talking about link prediction as an analog for recommender systems or I guess it is one possible way to do recommendations

Kyle Polich11:05And it seems like we can benefit from all of the existing approaches to link prediction But what about something like node similarity that uh we might identify you as a customer and uh do you think you know if you and I were in uh embarking on a new venture we had no research history to go on we've got to build a from the ground up new recommender system

Kyle Polich11:26And we've got to decide where to invest our R&D Obviously we're going to put some in the link prediction What do you think about node similarity as a path towards recommenders

Asaf Shapira11:35I

Asaf Shapira11:35think what what you were talking about is the cold start problem So you you don't have it yes

Kyle Polich11:41yes exactly

Asaf Shapira11:42So you don't have any similar any nodes to compare between

Kyle Polich11:46So no I might know properties of a node before I see all of its links Uh for example uh I might well I have its IP address so from that I know some geographic information I could extend that with whatever you know maybe I know you from some sort of shady third party tracking you know but I know things about a node before I know all its edges

Asaf Shapira12:07If you you get you get it you get it from attributes that aren't network related OK so that's why you interview other people because I'm a network guy

Asaf Shapira12:18Fair enough

Kyle Polich12:21Oh you'd mentioned that the students you're helping uh are embarking on a recommender system project What made that an ideal thing for them to study

Asaf Shapira12:30Actually I I didn't initiate the project A colleague of mine initiate got the let's say requests from the industry for

Asaf Shapira12:39Projects and then uh she referred them to her students Uh shout out to Yolit uh from uh from uh Shinka College What they needed was a recommendation system

Asaf Shapira12:53But because they have sensitive data they didn't want it to you know to run on the text to see text or labels and so on What I suggested is to use network properties in order to create a recommendation system so we don't need to see the labels We don't need to see the

Asaf Shapira13:16Data itself only the connections and by anonymizing the data we can create a network project it or something like that and give a score for a recommendation

Kyle Polich13:27What do you have personal experience with recommender systems Do they do you find that they delight you or annoy you

Asaf Shapira13:33I guess I I sometimes feel I hit I kind of hit a glass ceiling you know where there's a

Asaf Shapira13:39When they recommend me the same things and

Kyle Polich13:43you looked at flowers the day before Mother's Day you don't need to see them six months

Kyle Polich13:46later

Asaf Shapira13:47Let's imagine I'm a random walker OK I'm traveling through my community and I discover all kinds of things OK and then I want some teleporting I want I want the recommendation system to take me somewhere I didn't know I need to go and discover me something new that I would like you know that's the hardest thing because

Asaf Shapira14:07If I'm thinking from a network point of view you know 80% is within communities The strong the strong connections most of the connections are within community and the weak connections the connections that take us to a different place are rare

Asaf Shapira14:24Weak connections that take us to an interesting place a place that interests us are even rarer So that's the problem So my solution is uh in the case of let's say uh streaming I do uh let's say a Netflix um I I have a Netflix account

Asaf Shapira14:41A month later I cancel it and do an Amazon account and then you know I do a I'm circling around so that's the random walker of streaming So that's my tip of the day except for Disney that's my children won't let me cancel it

Kyle Polich14:56Well I've got a 3rd

Kyle Polich14:57Maybe my 3rd and final thought about how we could apply network science We talked about link prediction maybe node similarity What about something from influencer networks or social networks like using PageRank For example I see a lot of books coming out that are sort of about network theory but some of them are just sort of soft armchair books like a psychologist wrote it

Kyle Polich15:19And it's not necessarily my cup of tea but if you recommended a book on network science that as an authority in the area would convey uh I would think a lot of weight towards a good recommendation for someone who likes the technical literature and the scientific literature

Asaf Shapira15:34Yes that's the whole idea behind the pagering right It was created in order to recommend on websites And by the way I know which book you're

Asaf Shapira15:44talking about

Kyle Polich15:44Oh then enlighten me I'm actually not sure I've seen more than one sort of soft book

Asaf Shapira15:49I thought you were talking about Malcolm Gladwell's book about uh oh I think it was the Tipping Point I think he wrote it

Kyle Polich15:56Yeah

Asaf Shapira15:58The interesting thing is that he hasn't mentioned the word network once in this book but he's actually talking about the Greece centrality between the centrality and so on I can prove why this book is is wrong but maybe for another episode or

Kyle Polich16:15that I would love to do that episode if you want to do a takedown

Asaf Shapira16:18OK

Asaf Shapira16:20Well head to head with Malcolm Gladwell Let's see

Kyle Polich16:25I

Kyle Polich16:25should read it in between then So let me make a note I haven't read it yet Oh

Asaf Shapira16:29you know he writes I think he apologized for this book like 100 times since then but for for different reasons not network reasons but

Asaf Shapira16:39The problem with this book I'll just say it in a few words is when he talks about centrality he actually talks about personality You can have all kinds of persons right personas but the thing is that even if you're let's say outgoing and so on it doesn't mean you're an influencer It probably says that you have maybe a high out degrees

Asaf Shapira16:58But it doesn't say that you it doesn't necessarily say that you have a high in degree When you build your theory of influence on personas personas are probably normally distributed right So you should have a few like this Most of us are let's say average

Asaf Shapira17:17But we know influencers are not normally distributed right They are uh it's a long tail so by definition personas aren't supposed to be a big part of what it is to be uh an influencer

Kyle Polich17:32Well it's gonna be one of the challenges I hope to talk to some researchers about as we start exploring more into the recommender systems So thank you for your advice along the way