research-article

“Who doesn’t like dinosaurs?” Finding and Eliciting Richer Preferences for Recommendation

Authors:

Tobias Schnabel,

Saleema AmershiAuthors Info & Claims

RecSys '20: Proceedings of the 14th ACM Conference on Recommender Systems

Pages 398 - 407

https://doi.org/10.1145/3383313.3412267

Published: 22 September 2020 Publication History

Abstract

Real-world recommender systems often allow users to adjust the presented content through a variety of preference elicitation techniques such as “liking” or interest profiles. These elicitation techniques trade-off time and effort to users with the richness of the signal they provide to learning component driving the recommendations. In this paper, we explore this trade-off, seeking new ways for people to express their preferences with the goal of improving communication channels between users and the recommender system. Through a need-finding study, we observe the patterns in how people express their preferences during curation task, propose a taxonomy for organizing them, and point out research opportunities. We present a case study that illustrates how using this taxonomy to design an onboarding experience can lead to more accurate machine-learned recommendations while maintaining user satisfaction under low effort.

References

[1]

Xavier Amatriain, Josep M Pujol, Nava Tintarev, and Nuria Oliver. 2009. Rate it again: increasing recommendation accuracy by user re-rating. In Proceedings of the third ACM conference on Recommender systems. ACM, 173–180.

Digital Library

[2]

Naveen Farag Awad and Mayuram S Krishnan. 2006. The personalization privacy paradox: an empirical evaluation of information transparency and the willingness to be profiled online for personalization. MIS quarterly (2006), 13–28.

Digital Library

[3]

Lee Roy Beach. 1993. Broadening the definition of decision making: The role of prechoice screening of options. Psychological Science 4, 4 (1993), 215–220.

[4]

Toine Bogers and Marijn Koolen. 2017. Defining and supporting narrative-driven recommendation. In RecSys. 238–242.

[5]

Carolyn Brodie, Clare-Marie Karat, and John Karat. 2004. Creating an E-commerce environment where consumers are willing to share personal information. In Designing personalized user experiences in eCommerce. Springer, 185–206.

[6]

John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013. Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociological Methods & Research 42, 3 (2013), 294–320.

[7]

Ben Carterette, Paul N Bennett, David Maxwell Chickering, and Susan T Dumais. 2008. Here or there. In European Conference on Information Retrieval. Springer, 16–27.

[8]

Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards conversational recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 815–824.

Digital Library

[9]

Bruce Ferwerda, Emily Yang, Markus Schedl, and Marko Tkalcic. 2015. Personality Traits Predict Music Taxonomy Preferences. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems(CHI EA ’15). ACM, New York, NY, USA, 2241–2246. https://doi.org/10.1145/2702613.2732754

Digital Library

[10]

Samuel D Gosling, Peter J Rentfrow, and William B Swann Jr. 2003. A very brief measure of the Big-Five personality domains. Journal of Research in personality 37, 6 (2003), 504–528.

[11]

Tom Hope and Dafna Shahaf. 2018. Ballpark crowdsourcing: The wisdom of rough group comparisons. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 234–242.

Digital Library

[12]

J.A. Howard. 1977. Consumer behavior: application of theory. McGraw-Hill.

[13]

Rong Hu and Pearl Pu. 2009. A Comparative User Study on Rating vs. Personality Quiz Based Preference Elicitation Methods. In Proceedings of the 14th International Conference on Intelligent User Interfaces(IUI ’09). ACM, New York, NY, USA, 367–372. https://doi.org/10.1145/1502650.1502702

Digital Library

[14]

Rong Hu and Pearl Pu. 2011. Enhancing collaborative filtering systems with personality information. In Proceedings of the fifth ACM conference on Recommender systems. ACM, 197–204.

Digital Library

[15]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In ICDM. 263–272.

[16]

Jie Kang, Kyle Condiff, Shuo Chang, Joseph A Konstan, Loren Terveen, and F Maxwell Harper. 2017. Understanding how people use natural language to ask for recommendations. In Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 229–237.

Digital Library

[17]

Diane Kelly and Jaime Teevan. 2003. Implicit feedback for inferring user preference: a bibliography. In SIGIR Forum, Vol. 37. 18–28.

[18]

Daniel Kluver, Tien T Nguyen, Michael Ekstrand, Shilad Sen, and John Riedl. 2012. How many bits per rating?. In Proceedings of the sixth ACM conference on Recommender systems. ACM, 99–106.

Digital Library

[19]

Bart P Knijnenburg, Niels JM Reijmer, and Martijn C Willemsen. 2011. Each to his own: how different users call for different interaction methods in recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. ACM, 141–148.

Digital Library

[20]

Bart P Knijnenburg, Martijn C Willemsen, and Stefan Hirtbach. 2010. Receiving recommendations and providing feedback: The user-experience of a recommender system. In International Conference on Electronic Commerce and Web Technologies. Springer, 207–216.

[21]

Shyong Lam, Dan Frankowski, and John Riedl. 2006. Do you trust your recommendations? An exploration of security and privacy issues in recommender systems. Emerging trends in information and communication security (2006), 14–29.

[22]

Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommendations. In Advances in Neural Information Processing Systems. 9725–9735.

[23]

Ting Li and Till Unger. 2012. Willing to pay for quality personalization? Trade-off between quality and privacy. European Journal of Information Systems 21, 6 (2012), 621–642.

[24]

Yabing Liu, Krishna P Gummadi, Balachander Krishnamurthy, and Alan Mislove. 2011. Analyzing facebook privacy settings: user expectations vs. reality. In Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference. ACM, 61–70.

Digital Library

[25]

Robert R. McCrae and Oliver P. John. 1992. An Introduction to the Five-Factor Model and Its Applications. Journal of Personality 60, 2 (1992), 175–215. https://doi.org/10.1111/j.1467-6494.1992.tb00970.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-6494.1992.tb00970.x

[26]

Sean M McNee, Shyong K Lam, Joseph A Konstan, and John Riedl. 2003. Interfaces for eliciting new user preferences in recommender systems. In International Conference on User Modeling. Springer, 178–187.

[27]

Hien Nguyen and Peter Haddawy. 1999. The decision-theoretic interactive video advisor. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., 494–501.

[28]

Tien T Nguyen, Daniel Kluver, Ting-Yu Wang, Pik-Mai Hui, Michael D Ekstrand, Martijn C Willemsen, and John Riedl. 2013. Rating support interfaces to improve user experience and recommender accuracy. In Proceedings of the 7th ACM conference on Recommender systems. ACM, 149–156.

Digital Library

[29]

Syavash Nobarany, Louise Oram, Vasanth Kumar Rajendran, Chi-Hsiang Chen, Joanna McGrenere, and Tamara Munzner. 2012. The design space of opinion measurement interfaces: exploring recall support for rating and ranking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2035–2044.

Digital Library

[30]

David M Pennock, Eric Horvitz, Steve Lawrence, and C Lee Giles. 2000. Collaborative filtering by personality diagnosis: A hybrid memory-and model-based approach. In Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence. 473–480.

[31]

Alina Pommeranz, Joost Broekens, Pascal Wiggers, Willem-Paul Brinkman, and Catholijn M Jonker. 2012. Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process. User Modeling and User-Adapted Interaction 22, 4-5 (2012), 357–397.

Digital Library

[32]

Al Mamunur Rashid, Istvan Albert, Dan Cosley, Shyong K Lam, Sean M McNee, Joseph A Konstan, and John Riedl. 2002. Getting to know you: learning new user preferences in recommender systems. In Proceedings of the 7th international conference on Intelligent user interfaces. ACM, 127–134.

Digital Library

[33]

Paul Resnick and Hal R Varian. 1997. Recommender systems. Commun. ACM 40, 3 (1997), 56–59.

Digital Library

[34]

C. J. Van Rijsbergen. 1979. Information Retrieval(2nd ed.). Butterworth-Heinemann.

[35]

Andrew I Schein, Alexandrin Popescul, Lyle H Ungar, and David M Pennock. 2002. Methods and metrics for cold-start recommendations. In SIGIR. 253–260.

[36]

Tobias Schnabel, Paul N Bennett, Susan T Dumais, and Thorsten Joachims. 2016. Using shortlists to support decision making and improve recommender system performance. In Proceedings of the 25th International Conference on World Wide Web. 987–997.

Digital Library

[37]

A.L. Strauss and J.M. Corbin. 1990. Basics of qualitative research: grounded theory procedures and techniques. Sage Publications. https://books.google.com/books?id=nvwOAQAAMAAJ

[38]

Mingxuan Sun, Fuxin Li, Joonseok Lee, Ke Zhou, Guy Lebanon, and Hongyuan Zha. 2013. Learning multiple-question decision trees for cold-start recommendation. In Proceedings of the sixth ACM international conference on Web search and data mining. 445–454.

Digital Library

[39]

Kristen Vaccaro, Dylan Huang, Motahhare Eslami, Christian Sandvig, Kevin Hamilton, and Karrie Karahalios. 2018. The Illusion of Control: Placebo Effects of Control Settings. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, Article 16, 13 pages. https://doi.org/10.1145/3173574.3173590

Digital Library

Cited By

Haruyama MHidaka K(2024)What influences users to provide explicit feedback? A case of food delivery recommendersUser Modeling and User-Adapted Interaction10.1007/s11257-023-09385-834:3(753-796)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s11257-023-09385-8
Ziegler JLoepp B(2024)Interactive Recommendation SystemsHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_54-1(1-29)Online publication date: 11-Feb-2024
https://doi.org/10.1007/978-3-319-27648-9_54-1
Cao TBogers T(2024)“If I Like BLANK, What Else Will I Like?”: Analyzing a Human Recommendation Community on RedditWisdom, Well-Being, Win-Win10.1007/978-3-031-57850-2_6(70-83)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57850-2_6
Show More Cited By

Recommendations

Effects of Personalized and Aggregate Top-N Recommendation Lists on User Preference Ratings

Prior research has shown a robust effect of personalized product recommendations on user preference judgments for items. Specifically, the display of system-predicted preference ratings as item recommendations has been shown in multiple studies to bias ...
Eliciting pairwise preferences in recommender systems
RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

Preference data in the form of ratings or likes for items are widely used in many Recommender Systems. However, previous research has shown that even item comparisons, which generate pairwise preference data, can be used to model user preferences. ...
Preference recommendation for personalized search

Conditional Preference Networks (CP-nets) are widely used to express qualitative preferences. As users are sometimes reluctant or unable to specify complete CP-nets, it prohibits personalized search from being effectively conducted. In this article, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '20: Proceedings of the 14th ACM Conference on Recommender Systems

September 2020

796 pages

ISBN:9781450375832

DOI:10.1145/3383313

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

RecSys '20

Sponsor:

RecSys '20: Fourteenth ACM Conference on Recommender Systems

September 22 - 26, 2020

Virtual Event, Brazil

Acceptance Rates

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
664
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Haruyama MHidaka K(2024)What influences users to provide explicit feedback? A case of food delivery recommendersUser Modeling and User-Adapted Interaction10.1007/s11257-023-09385-834:3(753-796)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s11257-023-09385-8
Ziegler JLoepp B(2024)Interactive Recommendation SystemsHandbook of Human Computer Interaction10.1007/978-3-319-27648-9_54-1(1-29)Online publication date: 11-Feb-2024
https://doi.org/10.1007/978-3-319-27648-9_54-1
Cao TBogers T(2024)“If I Like BLANK, What Else Will I Like?”: Analyzing a Human Recommendation Community on RedditWisdom, Well-Being, Win-Win10.1007/978-3-031-57850-2_6(70-83)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1007/978-3-031-57850-2_6
Musto CMartina AIovine ANarducci Fde Gemmis MSemeraro G(2023)Tell me what you Like: introducing natural language preference elicitation strategies in a virtual assistant for the movie domainJournal of Intelligent Information Systems10.1007/s10844-023-00835-862:2(575-599)Online publication date: 12-Dec-2023
https://doi.org/10.1007/s10844-023-00835-8
Apostolidis KMezaris VPapadogiorgaki MBei ELivanos GZervakis M(2022)Content and Other Resources Recommendations for Individuals with Intellectual Disability: A ReviewElectronics10.3390/electronics1121347211:21(3472)Online publication date: 26-Oct-2022
https://doi.org/10.3390/electronics11213472
Kang HQian XHope TShahaf DChan JKittur A(2022)Augmenting Scientific Creativity with an Analogical Search EngineACM Transactions on Computer-Human Interaction10.1145/353001329:6(1-36)Online publication date: 16-Nov-2022
https://dl.acm.org/doi/10.1145/3530013
Chen QSchnabel TNushi BAmershi S(2022)HINT: Integration Testing for AI-based features with Humans in the LoopProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511141(549-565)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511141
Kim HLim Y(2021)Teaching-Learning Interaction: A New Concept for Interaction Design to Support Reflective User Agency in Intelligent SystemsProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3462141(1544-1553)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3462141
Musto CMartina AIovine Ade Gemmis MSemeraro G(undefined)Tell Me What You Like: Introducing Natural Language Preference Elicitation Strategies in a Virtual Assistant for the Movie DomainSSRN Electronic Journal10.2139/ssrn.4140047
https://doi.org/10.2139/ssrn.4140047

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten