Efficient Exploration and Exploitation for Sequential Music Recommendation
Authors: 
Bruno Laporais Pereira, Pedro Dalla Vecchia Chaves, Rodrygo L. T. SantosAuthors Info & Claims
Bruno Laporais Pereira, Pedro Dalla Vecchia Chaves, Rodrygo L. T. SantosAuthors Info & ClaimsArticle No.: 28, Pages 1 - 23
Abstract
Music streaming services heavily rely upon recommender systems to acquire, engage, and retain users. One notable component of these services are playlists, which can be dynamically generated in a sequential manner based on the user’s feedback during a listening session. Online learning to rank approaches have recently been shown effective at leveraging such feedback to learn users’ preferences in the space of song features. Nevertheless, these approaches can suffer from slow convergence as a result of their random exploration component and their session-agnostic exploitation component. To overcome these limitations, we propose a novel online learning to rank approach which efficiently explores the space of candidate recommendation models by restricting itself to the orthogonal complement of the subspace of previous underperforming exploration directions. Moreover, we propose a session-aware exploitation component which leverages the momentum of the current best model during updates. Our thorough evaluation using simulated listening sessions from two large Last.fm datasets demonstrates substantial improvements over state-of-the-art approaches in terms of early-stage performance, which results in an improved user experience during online learning. In addition, we demonstrate that long-term convergence can be further enhanced by adaptively relaxing exploration constraints along the way.
References
[1]
Himan Abdollahpouri. 2019. Popularity bias in ranking and recommendation. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM, New York, NY, 529–530. DOI:
[2]
Claudio Baccigalupo and Enric Plaza. 2006. Case-based sequential ordering of songs for playlist recommendation. In Proceedings of the European Conference on Case-Based Reasoning, Thomas R. Roth-Berghofer, Mehmet H. Göker, and H. Altay Güvenir (Eds.). Springer, Berlin, 286–300.
[3]
James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning. Sanjoy Dasgupta and David McAllester (Eds.), PMLR, Vol. 28, Atlanta, GA, 115–123.
[4]
Geoffray Bonnin and Dietmar Jannach. 2014. Automated generation of music playlists: Survey and experiments. ACM Computing Surveys 47, 2 (2014), 1–35.
[5]
Klaas Bosteels and Etienne E. Kerre. 2009. A fuzzy framework for defining dynamic playlist generation heuristics. Fuzzy Sets and Systems 160, 23 (2009), 3342–3358.
[6]
Oscar Celma. 2010. Music Recommendation and Discovery: The Long Tail, Long Fail, and Long Play in the Digital Music Space (1st ed.). Springer, Berlin.
[7]
Pedro Dalla Vecchia Chaves, Bruno L. Pereira, and Rodrygo L. T. Santos. 2022. Efficient online learning to rank for sequential music recommendation. In Proceedings of the ACM Web Conference 2022. ACM, New York, NY, 2442–2450. DOI:
[8]
Shuo Chen, Josh L. Moore, Douglas Turnbull, and Thorsten Joachims. 2012. Playlist prediction via metric embedding. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 714–722.
[9]
Yashar Deldjoo, Dietmar Jannach, Alejandro Bellogin, Alessandro Difonzo, and Dario Zanzonelli. 2023. Fairness in recommender systems: Research landscape and future directions. User Modeling and User-Adapted Interaction (2023), 1–50. https://link.springer.com/article/10.1007/s11257-023-09364-z#article-info
[10]
Ricardo Dias, Daniel Gonçalves, and Manuel J. Fonseca. 2017. From manual to assisted playlist creation: A survey. Multimedia Tools and Applications 76, 12 (2017), 14375–14403.
[11]
Bradley Efron. 1992. Bootstrap methods: Another look at the jackknife. In Breakthroughs in Statistics, Samuel Kotz and Norman L. Johnson (Eds.). Springer, New York, NY, 569–593.
[12]
Abraham D. Flaxman, Adam Tauman Kalai, and H. Brendan McMahan. 2005. Online convex optimization in the bandit setting: Gradient descent without a gradient. In Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms. Society for Industrial and Applied Mathematics, 385–394.
[13]
Don Gartner, Florian Kraft, and Thomas Schaaf. 2007. An adaptive distance measure for similarity based playlist generation. In Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing. Vol. 1, IEEE, I–229.
[14]
Claudio Gentile, Shuai Li, Purushottam Kar, Alexandros Karatzoglou, Giovanni Zappella, and Evans Etrue. 2017. On context-dependent clustering of bandits. In Proceedings of the 34th International Conference on Machine Learning. Vol. 70, PMLR, 1253–1262.
[15]
Artem Grotov and Maarten de Rijke. 2016. Online learning to rank for information retrieval: SIGIR 2016 tutorial. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 1215–1218.
[16]
Anja Nylund Hagen. 2015. The playlist experience: Personal playlists in music streaming services. Popular Music and Society 38, 5 (2015), 625–645.
[17]
Negar Hariri, Bamshad Mobasher, and Robin Burke. 2012. Context-aware music recommendation based on latenttopic sequential patterns. In Proceedings of the 6th ACM Conference on Recommender Systems. ACM, New York, NY, 131–138.
[18]
Negar Hariri, Bamshad Mobasher, and Robin Burke. 2015. Adapting to user preference changes in interactive recommendation. In Proceedings of the 24th International Conference on Artificial Intelligence. AAAI, New York, NY, 4268–4274.
[19]
David B. Hauver and James C. French. 2001. Flycasting: Using collaborative filtering to generate a playlist for online radio. In Proceedings of the International Conference on IEEE Web Delivering of Music. IEEE, 123–130.
[20]
Katja Hofmann. 2013. Fast and reliable online learning to rank for information retrieval. SIGIR Forum 47, 2 (2013), 140.
[21]
Katja Hofmann, Anne Schuth, Shimon Whiteson, and Maarten de Rijke. 2013. Reusing historical interaction data for faster online learning to rank for ir. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 183–192.
[22]
Binbin Hu, Chuan Shi, and Jian Liu. 2017. Playlist recommendation based on reinforcement learning. In Intelligence Science I, Springer International Publishing, Cham, 172-182.
[23]
Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In Proceedings of the 2008 8th IEEE International Conference on Data Mining. IEEE, 263–272.
[24]
Gawesh Jawaheer, Martin Szomszor, and Patty Kostkova. 2010. Comparison of implicit and explicit feedback from an online music recommendation service. In Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender Systems. ACM, New York, NY, 47–51.
[25]
Iman Kamehkhosh, Geoffray Bonnin, and Dietmar Jannach. 2020. Effects of recommendations on the playlist creation behavior of users. User Modeling and User-Adapted Interaction 30, 2 (2020), 285–322.
[26]
James King and Vaiva Imbrasaitė. 2015. Generating music playlists with hierarchical clustering and Q-learning. In Advances in Information Retrieval, Allan Hanbury, Gabriella Kazai, Andreas Rauber, and Norbert Fuhr (Eds.). Springer International Publishing, Cham, 315–326.
[27]
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web. ACM, New York, NY, 661–670.
[28]
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. 2011. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 297–306.
[29]
Elad Liebman, Maytal Saar-Tsechansky, and Peter Stone. 2015. DJ-MC: A Reinforcement-learning agent for music playlist recommendation. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems. ACM, 591–599.
[30]
Beth Logan. 2002. Content-based playlist generation: Exploratory experiments. In Proceedings of the 3rd International Conference on Music Information Retrieval. Vol. 2, 295–296.
[31]
Joshua L. Moore, Shuo Chen, Thorsten Joachims, and Douglas Turnbull. 2012. Learning to embed songs and tags for playlist prediction. In Proceedings of the 13th International Society for Music Information Retrieval Conference. 349–354.
[32]
Marta Moscati, Emilia Parada-Cabaleiro, Yashar Deldjoo, Eva Zangerle, and Markus Schedl. 2022. Music4All-onion – a large-scale multi-faceted content-centric music recommendation dataset. In Proceedings of the 31st ACM International Conference on Information Knowledge Management. ACM, 4339–4343. DOI:
[33]
Harrie Oosterhuis and Maarten de Rijke. 2017. Balancing speed and quality in online learning to rank for information retrieval. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, New York, NY, 277–286.
[34]
Harrie Oosterhuis and Maarten de Rijke. 2018. Differentiable unbiased online learning to rank. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management. ACM, New York, NY, 1293–1302.
[35]
Antti Oulasvirta, Janne P. Hukkinen, and Barry Schwartz. 2009. When more is less: The paradox of choice in search engine use. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 516–523.
[36]
Elias Pampalk, Tim Pohle, and Gerhard Widmer. 2005. Dynamic playlist generation based on skipping behavior. In Proceedings of the 6th International Conference on Music Information Retrieval. 634–637.
[37]
Snickars Pelle. 2017. More of the Same—On Spotify Radio. Culture Unbound. Journal of Current Cultural Research 9, 2 (2017), 184–211.
[38]
Bruno L. Pereira, Alberto Ueda, Gustavo Penha, Rodrygo L. T. Santos, and Nivio Ziviani. 2019. Online learning to rank for sequential music recommendation. In Proceedings of the 13th ACM Conference on Recommender Systems. ACM, New York, NY, 237–245.
[39]
Martin Pichl, Eva Zangerle, and Günther Specht. 2017. Understanding user-curated playlists on spotify: A machine learning approach. International Journal of Multimedia Data Engineering and Management 8, 4 (2017), 44–59.
[40]
Boris T. Polyak. 1964. Some methods of speeding up the convergence of iteration methods. Ussr Computational Mathematics and Mathematical Physics 4, 5 (1964), 1–17.
[41]
Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018. Sequence-aware recommender systems. ACM Computing Surveys 51, 4 (2018), 1–36.
[42]
Filip Radlinski, Madhu Kurup, and Thorsten Joachims. 2008. How does clickthrough data reflect retrieval quality? In Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 43–52.
[43]
Markus Schedl, Stefan Brandl, Oleg Lesota, Emilia Parada-Cabaleiro, David Penz, and Navid Rekabsaz. 2022. LFM-2b: A dataset of enriched music listening events for recommender systems research and fairness analysis. In Proceedings of the ACM SIGIR Conference on Human Information Interaction and Retrieval. ACM, New York, NY, 337–341.
[44]
Anne Schuth, Harrie Oosterhuis, Shimon Whiteson, and Maarten de Rijke. 2016. Multileave gradient descent for fast online learning to rank. In Proceedings of the 9th ACM International Conference on Web Search and Data Mining. ACM, New York, NY, 457–466.
[45]
Anne Schuth, Floor Sietsma, Shimon Whiteson, Damien Lefortier, and Maarten de Rijke. 2014. Multileaved comparisons for fast online evaluation. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. ACM, New York, NY, 71–80.
[46]
Andreu Vall. 2015. Listener-inspired automated music playlist generation. In Proceedings of the 9th ACM Conference on Recommender Systems. ACM, New York, NY, 387–390.
[47]
Andreu Vall, Matthias Dorfer, Markus Schedl, and Gerhard Widmer. 2018. A hybrid approach to music playlist continuation based on playlist-song membership. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing. ACM, New York, NY, 1374–1382.
[48]
Maksims Volkovs and Guang Wei Yu. 2015. Effective latent models for binary feedback in recommender systems. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 313–322.
[49]
Huazheng Wang, Sonwoo Kim, Eric McCord-Snook, Qingyun Wu, and Hongning Wang. 2019. Variance reduction in gradient exploration for online learning to rank. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 835–844.
[50]
Huazheng Wang, Ramsey Langley, Sonwoo Kim, Eric McCord-Snook, and Hongning Wang. 2018. Efficient exploration of gradient space for online learning to rank. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, New York, NY, 145–154.
[51]
Shoujin Wang, Longbing Cao, Yan Wang, Quan Z. Sheng, Mehmet A. Orgun, and Defu Lian. 2021. A survey on session-based recommender systems. ACM Computing Surveys 54, 7, Article 154 (July 2021), 38 pages. DOI:
[52]
Xuanhui Wang, Nadav Golbandi, Michael Bendersky, Donald Metzler, and Marc Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the 11th ACM International Conference on Web Search and Data MiningACM, New York, NY, 610–618. DOI:
[53]
Xinxi Wang, Yi Wang, David Hsu, and Ye Wang. 2014. Exploration in interactive personalized music recommendation: A reinforcement learning approach. ACM Transactions on Multimedia Computing, Communications, and Applications 11, 1 (2014), 1–22.
[54]
Zhe Xing, Xinxi Wang, and Ye Wang. 2014. Enhancing collaborative filtering music recommendation by balancing exploration and exploitation. In Proceedings of the 15th International Society for Music Information Retrieval Conference. 445–450.
[55]
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of the International Conference on Learning Representations.
[56]
Yisong Yue and Thorsten Joachims. 2009. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning. ACM, New York, NY, 1201–1208.
[57]
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021. Optimizing dense retrieval model training with hard negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 1503–1512. DOI:
[58]
Tong Zhao and Irwin King. 2016. Constructing reliable gradient exploration for online learning to rank. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, New York, NY, 1643–1652.
[59]
Shengyao Zhuang and Guido Zuccon. 2021. How do online learning to rank methods adapt to changes of intent? In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 911–920. DOI:
Index Terms
- Efficient Exploration and Exploitation for Sequential Music Recommendation
Recommendations
Efficient Online Learning to Rank for Sequential Music Recommendation
WWW '22: Proceedings of the ACM Web Conference 2022Music streaming services heavily rely upon recommender systems to acquire, engage, and retain users. One notable component of these services are playlists, which can be dynamically generated in a sequential manner based on the user’s feedback during a ...
Online learning to rank for sequential music recommendation
RecSys '19: Proceedings of the 13th ACM Conference on Recommender SystemsThe prominent success of music streaming services has brought increasingly complex challenges for music recommendation. In particular, in a streaming setting, songs are consumed sequentially within a listening session, which should cater not only for ...
Exploration in Interactive Personalized Music Recommendation: A Reinforcement Learning Approach
Current music recommender systems typically act in a greedy manner by recommending songs with the highest user ratings. Greedy recommendation, however, is suboptimal over the long term: it does not actively gather information on user preferences and ...
Comments
Information & Contributors
Information
Published In
December 2024
210 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 31 July 2024
Online AM: 27 September 2023
Accepted: 11 September 2023
Revised: 28 August 2023
Received: 06 November 2022
Published in TORS Volume 2, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- CNPq and FAPEMIG
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 317Total Downloads
- Downloads (Last 12 months)216
- Downloads (Last 6 weeks)9
Reflects downloads up to 29 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in