DataRec Library for Reproducible in Recommend Systems
In this episode of Data Skeptic's Recommender Systems series, host Kyle Polich explores DataRec, a new Python library designed to bring reproducibility and standardization to recommender systems research. Guest Alberto Carlo Maria Mancino, a postdoc researcher from Politecnico di Bari, Italy, discusses the challenges of dataset management in recommendation research—from version control issues to preprocessing inconsistencies—and how DataRec provides automated downloads, checksum verification, and standardized filtering strategies for popular datasets like MovieLens, [Last.fm](http://Last.fm), and Amazon reviews. The conversation covers Alberto's research journey through knowledge graphs, graph-based recommenders, privacy considerations, and recommendation novelty. He explains why small modifications in datasets can significantly impact research outcomes, the importance of offline evaluation, and DataRec's vision as a lightweight library that integrates with existing frameworks rather than replacing them. Whether you're benchmarking new algorithms or exploring recommendation techniques, this episode offers practical insights into one of the most critical yet overlooked aspects of reproducible ML research.
Guest
Alberto Carlo Maria Mancino: I am Alberto Carlo Maria Mancino, a post-doctoral researcher in Artificial Intelligence. You can find me at the Politecnico di Bari, Italy. My research focuses on Recommender Systems, particularly in studying knowledge-aware recommenders, graph-based recommenders, and the impact of data characteristics on recommenders' privacy and robustness.
Transcript
Kyle Polich : Welcome to Data Skeptic a podcast exploring the methods use cases and consequences of recommender systems
Kyle Polich : Welcome to another installment of Data Skeptic Recommender System In this episode we're putting the spotlight on a somewhat new Python library called Data Wreck It aims to provide a standard tool sets for reproducible analysis I've been covering reproducibility on the show since probably just about the very beginning
Kyle Polich : And Data rec takes those values and brings them specifically to the recommender system world
Kyle Polich : It'll help you with data set management versioning and making everything a lot more traceable
Kyle Polich : So if for example you're working on a new approach a new methodology and you want a benchmark I know of no other better tool to get standardized data sets to try your methodology in a variety of contexts
Kyle Polich : Maybe you've got your own data but you want to split it and clean it and things like that in a nice way That's yet one more of the features Data Rec has available for you Maybe you wanna head over to their GitHub page while you hear the interview Either way let's jump right into that
Alberto Carlo Mario Mancino : I am Alberto Carlo Mario Mancino from the CCM Flow Laboratory at the Pettericodi Bari in the south of Italy And basically I'm a postdoc researcher that devoted his whole research career on recommended systems So happy to be here talking about the recommended systems
Kyle Polich : So where does that story begin for you Is it as an undergrad or something else
Alberto Carlo Mario Mancino : Yeah it started with my master's thesis So I started studying recognition system in my with my master's thesis here at the Laboratory and now we do recognition systems It's a why So I found this very nice environment to learn to study recognition systems So I started with that and I'm still continuing
Alberto Carlo Mario Mancino : on
Kyle Polich : And so that's been a good amount of time to see the field and I guess a lot has evolved in that time as well How have you seen the field change
Alberto Carlo Mario Mancino : You know we have to mention LLMs if if we want to talk about evolutions in the systems research And before LLMs we had a lot of novelties because inco systems you can basically apply all the novelties that come from machine learning research So if there are new architectures new ways to export feature
Alberto Carlo Mario Mancino : Construction and so on you can always find a way to develop them in record systems and exploit them to make them even greater At some point in the in the in the previous years we had like a situation where we we didn't have so many new technologies so many new architectures that changed
Alberto Carlo Mario Mancino : Everything We were you know just improving the already existing methods But with LLMs we had a new paradigm And so nowadays if you take a look at conferences on recommended systems you will find a lot of papers on LLMs Not all of them are about use LMs or generative models but you can find a lot of them
Kyle Polich : And could you say a few things about your specific areas of interest Recommender systems is a pretty big topic Where have you specialized or contributed yourself
Alberto Carlo Mario Mancino : I started with knowledge recommender systems so recommended systems where you use knowledge graphs so public knowledge graphs but you can also use private ones to improve the recommendation performance or to uh induce some specific knowledge
Alberto Carlo Mario Mancino : Moved to uh graph-based recommenders because graph-based recommenders can easily be merged with knowledge graphs so it's even more easier to exploit them And I also saw that in Data Skeptic you did a season on graphs and gens It was very interesting because I think that with graphs you can do a lot of things They are very very easy to manipulate and to do very very cool stuff on them
Alberto Carlo Mario Mancino : And recently I moved on um analyzing some specific uh aspects of recognition system For example we published a paper on the importance of privacy in recognition systems because you know you have to deal with user data So you need to be aware of the importance of protecting user privacy And also very recently at Rexis which is let let's say
Alberto Carlo Mario Mancino : The main conference on recommended systems we published a paper on the importance of novelty in the in the recommendations It means that you do not focus only on the accuracy of your recommendations how good they are but also how much diverse they are because you want to give to the user a very nice experience So you don't want the user to
Alberto Carlo Mario Mancino : To to consume the same kind of items every time but to let the user explore the overall catalog So as you said in record systems you can do a lot of different things So it's also easy to be attracted from all different kinds of novelties and interesting aspects So yeah I did uh very different topics in recommended system but they all are very cool to study
Kyle Polich : So you've covered a lot of ground then methodologically speaking along the way what sort of data sets can you rely on to do that work
Alberto Carlo Mario Mancino : Basically in the research what you can do is to do the so-called offline evaluation It means that you don't properly have a platform where you can test your recommended system Of course if you are in the industry and
Alberto Carlo Mario Mancino : Uh specifically in a company that deploys recommended systems you can do that You can do for example AB testing and so on But if you are not in a in a company like that if you are in a university you have to find a way to test your algorithms and basically you do that uh with offline evaluation It means that you have data sets that are already prepared and shared publicly shared
Alberto Carlo Mario Mancino : And on those data sets you do all your all all your tests all your evaluations you compare models strategies and as I said I I analyze different aspects of the of recommended systems I had to deal with different data sets with different ways for managing these data sets
Alberto Carlo Mario Mancino : So if you deal with recommended systems in research and in particular with offline evaluation you will see at some point that there are flaws in this process because data sets usually are copy and paste or uh authors do not refer always to the to the to to the right source uh for retrieving these data sets basically And what happens is that a smaller even a smaller
Alberto Carlo Mario Mancino : Modification in the input can have a large impact on the output and this can affect the performance of recommended systems and basically the comparison of the performance of different models which is the main way we use in the research to understand if a specific approach is good or not
Kyle Polich : So I hear the famous movie lens data set come up quite a bit Are there any other canonical ones that you found to be particularly popular as you did a literature search
Alberto Carlo Mario Mancino : Yeah there is for example Last FM It's a a data set for music recommendation
Alberto Carlo Mario Mancino : But also there is a collection of data sets retrieved from the Amazon platform So you have a lot of data sets extracted from the Amazon platform So by taking the reviews of the users so you have the stars you also have the comments
Alberto Carlo Mario Mancino : So you can also do a review-based recommendation which means that you process the review to find a strategy that can lead you to recommend good items for the user based on the on his or her reviews So you have you have them you have Gala which is another data that is a point of interest dataset You have a lot of them
Alberto Carlo Mario Mancino : There are a lot of data sets that you can use Sometimes it's also difficult to compare results within different papers because one paper used one kind of data sets another one used different kinds of data sets This means that you cannot compare directly the performance of the models You have to rerun the experiments and by rerunning the experiments sometimes
Alberto Carlo Mario Mancino : Something can change Some something can be different in the configuration Sometimes even changing the computer on which we are running the experiments can affect the final performance
Kyle Polich : So with all those data sets to choose from if I was developing a new method I would want to try it on all of them or as many as I could but I get the sense they're not all a clean ready to go data set I've got to do some pre-processing first What does that typically consist of
Alberto Carlo Mario Mancino : In in recommended systems usually you don't use the data set as is because recommendation is not a standard machine learning task this data that are very very sparse
Alberto Carlo Mario Mancino : For each user you have just a bunch of interactions It can be like in the order of 1012 100 It's very difficult to have more than that
Alberto Carlo Mario Mancino : With just that limited number of the of of interactions you need to predict
Alberto Carlo Mario Mancino : Uh which kind of items the user will like by facing a catalog that can be huge can be very very very big very very large So for example you can have like 20 20,000 items but these kind of numbers are the numbers that we use in in research In real life Think about Amazon for example I don't know how many products they they have but they they are for sure in the range of billions of items So just
Alberto Carlo Mario Mancino : By looking at a very small set of interactions you have to predict what the user will like by facing a very large number of items Uh and so for this reason you need sometimes to do processing uh strategies like for example retaining users that have at least uh a specific number of interactions or discarding items that are very rare enjoyed by a very
Alberto Carlo Mario Mancino : Limited number of users because sometimes they are like noise in your dataset so you want to remove them But also there are filterings and rating strategies that are kind of a methodological um approach for the evaluation of your model For example you can do a temporal splitting or temporal uh filtering of your data set So you want to filter out all
Alberto Carlo Mario Mancino : Interactions that are before a specific time stamp or after another one in order to uh retain the consequentiality of the events So for example if I do a temporal splitting what I expect from my data set is that in the training set I have events that come before a specific time stamp And in the validation uh set and in the test set I have items that come after the time stamps
Alberto Carlo Mario Mancino : This kind of filterings and uh splitting are a way to reflect real life behaviors real life user engagement with the platform that still we cannot totally replicate
Alberto Carlo Mario Mancino : Because of course it's not a real online evaluation it's an offline evaluation so it's a simulation and yeah that's it Basically uh this is what you you can you can find in the when you have to deal with prep processing recommendation data
Kyle Polich : Well could you introduce Data rec and how this open source Python library might help me balance all those intricate needs and build something a little bit more reproducible
Alberto Carlo Mario Mancino : Datarec as you said is is a Python library So the idea is to make something that is plug and play in your in your scripts Basically with Datarec to have a library that allows you to retrieve the correct version of the data set and apply all the filterings and splitting that you need to run your experiments
Alberto Carlo Mario Mancino : Data that derives from my experience honestly from uh all the problems that I that I faced when dealing with recommendation data sets for offline evaluation
Alberto Carlo Mario Mancino : It burned like naturally you know I collected my own scripts and my own Python code to manipulate data set Then study by study paper by paper this kind of code repository became more and more complete with more modular and so on And so at some point my coauthors and I said OK why not share this with other researchers
Alberto Carlo Mario Mancino : We started by analyzing the literature and see which are the most used data sets OK so let's focus on them because if you want to consider all of them you have to consider hundreds of data sets So it makes everything you know more sparse more harder to deal with it And then we moved on the filtering strategies Let's focus on the most used ones And so also for the splitting strategies and so on
Alberto Carlo Mario Mancino : And then once we we framed our library based on these studies we started implementing all of them just make this library easy to use We are still working on it of course because when you have to manage a Python library you have to constantly update it uh check for bugs but also we are continuing uh by integrating useful data sets useful filterings and so on
Alberto Carlo Mario Mancino : Let's try to mitigate this scenario in the recommended system research where we have so many datasets but no one really knows every time where they should retrieve them download them how they should process them and so let's try to make at least this part standard
Alberto Carlo Mario Mancino : Even if of course people will propose new new strategies new ways to deal with it and that's totally OK but still let's uh everyone agree on how we should download and link to these data sets
Alberto Carlo Mario Mancino : So what we did basically is like to reconstruct the history of each data set find the original source and make the library point only to that source So when you use data rack you can automatically download the data set You you don't have to do anything You don't have to copy and paste from anything You don't have to download the data from whatever Google Drive folder online You have just to run the code The code then downloads the data set There is a checksum check
Alberto Carlo Mario Mancino : So that when when the data is downloaded that data rack checks that the checksum is the same that the library knows to be sure that modifications didn't happen For example it it can be that the authors of the data sets that publish the data set can update the the
Alberto Carlo Mario Mancino : Data set and then at that point you don't know because maybe they uh they published it at the same link So from the same link you'd think that you are downloading the same dataset but it's not because the users updated that dataset and so on So this is the idea and the motivation that drove us to develop Data rack
Kyle Polich : So it sounds like if a researcher wanted to use it they could get a big head start save a lot of time get a more standardized version of the data set pre-cleaned and all that What is their deliverable though once they use it for example do they get a CSV or how do they get the data
Alberto Carlo Mario Mancino : So they come as they are from the original source So for example if the data is stored as a Jason dataset it is downloaded as it is on your machine but then Dare creates a data structure on that
Alberto Carlo Mario Mancino : Even if you are dealing with different data sets that are stored in different ways so you can have a TSV CSV you can have JSON data sets and so on
Alberto Carlo Mario Mancino : Then Datare provides you what we call readers for these different data sets and you have already prepared readers and classes for well-known data sets And then when you use the data rec library you see only a datare object which is the same for all the data sets that are already implemented
Alberto Carlo Mario Mancino : In our library so you can reuse the same pipelines the same splitting the same filterings on different data sets without changing your code You have just to change the the object that you are using So Datareck does this under the hood But before doing this Datare checks that everything is OK that the file is that
Alberto Carlo Mario Mancino : And so basically when we add new data sets on Datarec what we do is to identify the original source compute the checksum and then find the right way to transport this raw file in a data rack object so that you can use all the data rack utilities
Kyle Polich : DeleteM makes it easy quick and safe to remove your personal data online at a time when surveillance and data breaches are common enough to make everyone vulnerable
Kyle Polich : Delete me knows your privacy is worth protecting Sign up and provide DeleteM with exactly what information you want deleted and their experts take it from there
Kyle Polich : Delete me sends you regular personalized privacy reports showing what info they found where they found it and what they removed I keep hearing about data breaches in the news In a recent Coinbase data breach a hacker obtained names addresses phone numbers and the last four digits of users' Social Security numbers plus masked bank account numbers Data brokers can vacuum up a lot of data creating a searchable profile about you and they can sell it to whoever they want
Kyle Polich : Thankfully if your data is exposed in a breach DeleteM can help remove it from the data brokers sites Take control of your data and keep your private life private by signing up for Delete Me now at a special discount for our listeners Get 20% off your Delete Me plan when you go to joindeleteme.com/data and use the promo code data at checkout
Kyle Polich : The only way you get the 20% off is to go to joindeleteme.com/data Enter code data at checkout That's joindeleteme.com/datacodedata
Kyle Polich : Thanks to this week's sponsor Zapier You know these days it feels like everyone is talking about AI not just me
Kyle Polich : But if there's one thing you've learned from this show it's that talking about a new technology isn't the same as implementing it Zapier is where tech innovators break the hype cycle and put AI to work for real
Kyle Polich : I've been using Zapier for at least a decade building little automation workflows and and honestly they've become even more essential in the AI era On the podcast side we've built a bunch of publishing workflows Audio files need to be moved around the cloud Things need to be organized And on the business side we've built Zapier automations around scheduling new lead gen inbound you name it The productivity gain has been tremendous but it's more than that
Kyle Polich : It's about unlocking new opportunities I wouldn't have time for otherwise What I love about Zapier is that it's for everyone
Kyle Polich : Tech expert or not yeah I could probably code it myself but it's just faster to do it in zappier You can connect top AI models like Chat GPT and cloud
Kyle Polich : To the tools your team already uses so you can add AI exactly where you need it whether that's AI powered workflows an autonomous agent customer service chatbot or something else entirely You can orchestrate it with Zapier No IT bottlenecks no complexity no AI hype just results Teams have already automated over 300 million AI tasks using Zapier Are you ready to join them
Kyle Polich : Get started for free by visiting Zapier.com/data skeptic That's Z A P I E R.com/data skeptic
Kyle Polich : So what does it take for a researcher to get started I guess it's Pip install data rack maybe a step one what's step 2 through I don't know 45 or 6 perhaps
Alberto Carlo Mario Mancino : Yeah PIP install data rack it's my objective because still we we are doing some smaller modifications to the library and before publishing it on the package manager of Python we want to you know be sure that everything is there and works properly In the next future that's the way
Alberto Carlo Mario Mancino : Daterre it's a it's a library We want that to be lightweight so that you can use Dare in your own code We we wanted to provide a library for recommendation data sets And and this comes from also from the actual scenario in the recommendation research where we have a lot of so-called reproducibility frameworks
Alberto Carlo Mario Mancino : So in recommended system research we have frameworks which are like preconstructed pipelines where you can inject your own code So it's a different kind of approach So they don't give you utilities to integrate them in your code They give you an entire pipeline where you can
Alberto Carlo Mario Mancino : Inject your data set or your new model and so on So I'm not saying that that's not correct That that's a strategy but with Darek we want we wanted to do something different We want to do to implement a library OK So it's something that you're not forced to stick on it You can use it then you can also discard it if you if you don't want to use it anymore But that thing
Alberto Carlo Mario Mancino : Does not change your pipeline because also researchers sometimes want to uh implement very short snippets very fast because just sometimes you have an idea but you're not sure that that will work So you want to make some very fast tests based on that idea and see if it's reasonable or not And doing this on a preconstructed framework
Alberto Carlo Mario Mancino : With specific rules specific models and so on makes this approach harder So sometimes even if you want to test a very small modification you need a lot of time to do that on on on these frameworks which are good for doing a lot of experiments and reproducibility studies and so on But for plug and play code for snippets and so on I think that a library it's a better alternative
Kyle Polich : Well that checksum you were mentioning I could see a scenario in which researchers who make use of the library might want to publish that checksum to say hey this is you know like the fingerprint of my data set Do you have any advice or a vision for how researchers that use the library might you know publish something like that or something similar
Alberto Carlo Mario Mancino : Yeah we thought about that
Alberto Carlo Mario Mancino : I honestly I don't know if it's already published in this version of Data rack but we already have that What happens in in Data rack Every single modification every single filtering and splitting that you can do on your data set is tracraced when you use the data rack object And at the end of this process or whenever you want in this sequence of transformations of your data set you can export
Alberto Carlo Mario Mancino : A YAML configuration file that can be then input to Datarack in order to redo the same processes so to replicate exactly the same processes and update the same data set of course So this is how Darack works and we did that with checksums That's the way that
Alberto Carlo Mario Mancino : You can do that But my uh advice for researchers and uh enthusiastic on recommended systems that want to do this is if even if you're not using Gittarack is don't do this at every single transformation of your data set because computer can check some sometimes can be very uh expensive in terms terms
Alberto Carlo Mario Mancino : Of the time needed to compute it So just do this at the very beginning at the very end of all the stuff that you do on your data If all the steps that you do are correct so you fix the seed you check that everything is replicable then you will always obtain the same results starting from the from the same set
Alberto Carlo Mario Mancino : So that's that's basically my advice And even if sometimes this can be something that could be not so so important who who cares if there is a little modification in this data set maybe the results will will will be the same
Alberto Carlo Mario Mancino : This is not true um let's say unfortunately because we have to be very very aware of this problem and the community honestly is because in recognis systems and in particular in Rxis in the Rxis conference we have a specific track that is devoted to the reproducibility studies which means that in this research truck
Alberto Carlo Mario Mancino : Authors publish papers about uh strategies to be sure that the results are reproducible or they compare previous results published in previous papers and so on so the community is aware about the importance of reproducibility and yeah I think that with data direct we are just trying to make this even stronger
Kyle Polich : I know in the psychology world they have something known as the reproducibility crisis Would you say it's at the point where it's a crisis or what's the right adjective for the state of maturity and reproducibility for recommender systems
Alberto Carlo Mario Mancino : I don't want to say that it's a crisis because on the long term I think that good approaches will always come up will always be more used than other approaches that maybe are are not so good because they were affected by some modifications in the data set However
Alberto Carlo Mario Mancino : If we don't do research properly if we don't take all the steps that are needed to be sure that our experiments are reproducible then we are just slowing down the research because another researcher from another part of the world at that point
Alberto Carlo Mario Mancino : You will need to reproduce all the experiments Maybe it will happen that person will face differences in the numbers published in the papers and what that person obtains by replicating that experiment This means that that person then has to go inside your source code understand what's happening why the numbers are not the same
Alberto Carlo Mario Mancino : And this just slows down the research process Let's think about LLMs for example They have public benchmarks They have websites with a lot of numbers and but this makes it easier for uh when a new model uh is is published
Alberto Carlo Mario Mancino : It it makes it easier to understand if it's better or not than the than the than the previous ones Of course benchmarks cannot tell everything about your model but are a good proxy feature for understanding if your approach is good or not So I don't want to say that we have uh a proper crisis in uh recommended systems but it's like we are damaging ourselves if we don't do research properly
Kyle Polich : And where is data wreck at on the adoption curve
Alberto Carlo Mario Mancino : Oh yeah it's it's honestly it's a it's a very beginning We published Ditare in July Quite new Yeah yeah it's a newborn
Alberto Carlo Mario Mancino : At Rxis 2025 we will present a tutorial on Data rack so we will start spreading the news with our colleagues with other researchers And yeah so Datarack is in the early days of its life and of course we wish that it would be more and more used and adopted but honestly I wish that it would be useful for someone out there making everything faster and easier even for early researchers
Alberto Carlo Mario Mancino : I remember my first days as a PhD student and I faced so many data sets I didn't know where to start with them And so I wish that in the next future uh new researchers will should not just think about that They can just focus on things that are are more interesting and more uh impactful like proposing new strategies proposing new models finding new holes in existing approaches and so on
Kyle Polich : Well what is the future of the project
Alberto Carlo Mario Mancino : In this stage Dareck provides to users functions and methods to transform your data rack export it and make it easy to be used in existing frameworks
Alberto Carlo Mario Mancino : Because still these frameworks that are out there are still used by researchers So instead of proposing something that is like an antagonist of these frameworks we want to publish something that is that could be let's say collaborator of this framework So so far Dect allows you to export the dataset and then to be used in these frameworks What we want to do
Alberto Carlo Mario Mancino : For example we were starting to do this with the core map framework because the authors of this framework contacted us to just to do this We want to publish versions of dataact that are already included in these frameworks so that all the frameworks do not focus on the first part of the pipeline which is the data management
Alberto Carlo Mario Mancino : Data that can do that and then the frameworks can focus on the models and on the evaluations on the evaluation of the results which are are the two components of the recommendation pipeline that are also very hard to be to set up to be sure that they are produceible that they are uh sharable between different frameworks and so on
Alberto Carlo Mario Mancino : So the idea in the next future is to make data direct already implemented in existing frameworks and we for sure we will do this with a couple of frameworks We will see if it will happen with other ones
Kyle Polich : And if listeners want to get their hands dirty where should they go to get involved and get started with the library
Alberto Carlo Mario Mancino : We released a couple of usages examples so uh I think that if you want to understand how to use that take a look at that We are going to publish in the next days our repository documentation
Alberto Carlo Mario Mancino : Which I think it's uh it's something that is needed Well we couldn't do that before so we are going to publish it in the next days I think before Rxis because we wanted to be that online before Rxis where we will have the tutorial on Dare
Alberto Carlo Mario Mancino : Yeah so if you want to use that take a look at the examples and the documentation We try to make it as easy to use as possible Of course we will continue to improve it to upgrade it but yeah everything it's out there And if if still you don't you are not able to use it contact us We will for sure reach out to you
Kyle Polich : And is there anywhere listeners can follow you online
Alberto Carlo Mario Mancino : Yeah you can find me on Twitter uh you can search for Alberto Mancino I have a personal webpage so if you search Alberto Mancino on Google you can find that So yeah I'm pretty everywhere online so it's it's it's easy to find me
Kyle Polich : Alberto thank you so much for taking the time to come on and share your work
Alberto Carlo Mario Mancino : Thank you
Alberto Carlo Mario Mancino : Carl It was very very uh fun interesting and thank you again for your