A comparative user study on rating vs. personality quiz based preference elicitation methods

Reviewer: Franz J Kurfess

Generating recommendations for customers-whether books, movies, or dating prospects-is easier if the customers' preferences are captured accurately. This paper compares two Web sites that have different approaches for capturing preferences about movies: ratings and personality quizzes. While the study described in the paper has its limitations, the overall question is quite important, and Hu and Pu offer a good framework and starting point for future work. The user interfaces of both sites were modified to have a similar look and offer the same functionality, in order to avoid bias based on the organization and features of the visual information. The authors use three criteria to determine the efficacy of the elicitation: perceived accuracy, user effort, and user loyalty. All three were subjectively reported by the participants, with efforts by the researchers to prevent response bias. The research shows no significant difference in perceived recommendation accuracy, significant difference in perceived effort invested, and some difference in loyalty. Overall, the results are more favorable for the personality quiz, but the paper concludes that there is insufficient research data to clearly state which system is better. The paper does not explore systems in which users provide rating data passively, such as Amazon's system of recommendations based on purchases and page-viewed numbers, which does not require active user involvement in the recommendation process. Neither does it address recommendations based on the features of the products themselves. Nevertheless, Hu and Pu's study offers an interesting investigation of a topic that has immense practical relevance, with relatively little systematic evaluation. Online Computing Reviews Service