skip to main content
research-article

Enhancing Multi-View Smoothness for Sequential Recommendation Models

Published: 08 April 2023 Publication History

Abstract

Sequential recommendation models aim to predict the interested items to a user based on his historical behaviors. To train sequential recommenders, implicit feedback data is widely adopted since it is easier to obtain than explicit feedback data. In the setting of implicit feedback, a user’s historical behaviors can be characterized as a chronologically ordered sequence of interacted items. From a perspective of machine learning, the historical interaction sequence and the recommended items can be considered as context and label, respectively, which are usually in one-hot representations in the recommendation models.
However, due to the discrete nature, one-hot representations are hard to sufficiently reflect the underlying user preference, and might also contain noise from implicit feedback that will mislead the model training. To solve these issues, we propose a general optimization framework, Multi-View Smoothness (MVS), to enhance the smoothness of sequential recommendation models in both data representations and model learning. Specifically, with the help of a complementary model, we smooth and enrich the one-hot representations of contexts and labels to better depict the underlying user preference (i.e., context smoothness and label smoothness), and devise a model regularization strategy to enforce the neighborhood smoothness of the model itself (i.e., model smoothness). Based on these strategies, we design three regularizers to constrain and improve the training of sequential recommendation models. Extensive experiments on five datasets show that our approach is able to improve the performance of various base models consistently and outperform other regularization training methods.

References

[1]
Wanyu Chen, Pengjie Ren, Fei Cai, Fei Sun, and Maarten de Rijke. 2022. Multi-interest diversification for end-to-end sequential recommendation. ACM Transactions on Information Systems 40, 1 (2022), Article 20, 30 pages.
[2]
Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018. Sequential recommendation with user memory networks. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM’18). ACM New York, NY, 108–116.
[3]
Yongjun Chen, Jia Li, and Caiming Xiong. 2022. ELECRec: Training sequential recommenders as discriminators. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’22). ACM, New York, NY.
[4]
Yongjun Chen, Zhiwei Liu, Jia Li, Julian J. McAuley, and Caiming Xiong. 2022. Intent contrastive learning for sequential recommendation. In Proceedings of the ACM Web Conference 2022 (WWW’22). ACM, New York, NY, 2172–2182.
[5]
Mingyue Cheng, Fajie Yuan, Qi Liu, Shenyang Ge, Zhi Li, Runlong Yu, Defu Lian, Senchao Yuan, and Enhong Chen. 2021. Learning recommender systems with implicit feedback via soft target enhancement. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’21). ACM, New York, NY, 575–584.
[6]
Ziwei Fan, Zhiwei Liu, Yu Wang, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, and Philip S. Yu. 2022. Sequential recommendation via stochastic self-attention. In Proceedings of the ACM Web Conference 2022 (WWW’22). ACM, New York, NY, 2036–2047.
[7]
Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S. Yu. 2021. Continuous-time sequential recommendation with temporal graph collaborative transformer. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM’21). ACM, New York, NY, 433–442.
[8]
Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo. 2020. Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations. ACM Transactions on Information Systems 39, 1 (2020), 1–42.
[9]
Chris Finlay, Adam M. Oberman, and Bilal Abbasi. 2018. Improved robustness to adversarial examples using Lipschitz regularization of the loss. arXiv:1810.000953 (2018).
[10]
Yingbo Gao, Weiyue Wang, Christian Herold, Zijian Yang, and Hermann Ney. 2020. Towards a better understanding of label smoothing in neural machine translation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing (AACL/IJCNLP’20). 212–223.
[11]
Shu Guo, Quan Wang, Bin Wang, Lihong Wang, and Li Guo. 2017. SSE: Semantically smooth embedding for knowledge graphs. IEEE Transactions on Knowledge and Data Engineering 29, 4 (2017), 884–897.
[12]
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of the 26th International Conference on World Wide Web (WWW’17). 173–182.
[13]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based recommendations with recurrent neural networks. In Proceedings of the 4th International Conference on Learning Representations (ICLR’16).
[14]
Balázs Hidasi, Massimo Quadrana, Alexandros Karatzoglou, and Domonkos Tikk. 2016. Parallel recurrent neural network architectures for feature-rich session-based recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, New York, NY, 241–248.
[15]
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR abs/1503.02531 (2015).
[16]
Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. 2022. Learning vector-quantized item representation for transferable sequential recommenders. arXiv preprint arXiv:2210.12316 (2022).
[17]
Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD’22). 585–593.
[18]
Xiaowen Huang, Jitao Sang, Jian Yu, and Changsheng Xu. 2022. Learning to learn a cold-start sequential recommender. ACM Transactions on Information Systems 40, 2 (2022), Article 30, 25 pages.
[19]
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL’20). 2177–2190.
[20]
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in NLP. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL’07).
[21]
Jiarui Jin, Xianyu Chen, Weinan Zhang, Junjie Huang, Ziming Feng, and Yong Yu. 2022. Learn over past, evolve for future: Search-based time-aware recommendation with sequential behavior data. In Proceedings of the ACM Web Conference 2022 (WWW’22). ACM, New York, NY, 2451–2461.
[22]
Wang-Cheng Kang and Julian J. McAuley. 2018. Self-attentive sequential recommendation. In Proceedings of the IEEE International Conference on Data Mining (ICDM’18). IEEE, Los Alamitos, CA, 197–206.
[23]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR’15): Conference Track Proceedings.
[24]
Walid Krichene and Steffen Rendle. 2020. On sampled metrics for item recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’20). ACM, New York, NY, 1748–1757.
[25]
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2017. Adversarial examples in the physical world. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17): Workshop Track.
[26]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444.
[27]
Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural attentive session-based recommendation. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM’17). ACM, New York, NY, 1419–1428.
[28]
Linyang Li and Xipeng Qiu. 2021. TAVAT: Token-aware virtual adversarial training for language understanding. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI’21).
[29]
Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018. STAMP: Short-term attention/memory priority model for session-based recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’18). ACM, New York, NY, 1831–1839.
[30]
Chen Ma, Liheng Ma, Yingxue Zhang, Jianing Sun, Xue Liu, and Mark Coates. 2020. Memory augmented graph neural networks for sequential recommendation. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI’20). 5045–5052.
[31]
Takeru Miyato, Shin-Ichi Maeda, Masanori Koyama, and Shin Ishii. 2019. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 8 (2019), 1979–1993.
[32]
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. 2019. When does label smoothing help? In Advances in Neural Information Processing Systems 32. 4696–4705.
[33]
Adam M. Oberman and Jeff Calder. 2018. Lipschitz regularized deep neural networks converge and generalize. CoRR abs/1808.09540 (2018).
[34]
Xingyu Pan, Yushuo Chen, Changxin Tian, Zihan Lin, Jinpeng Wang, He Hu, and Wayne Xin Zhao. 2022. Multimodal meta-learning for cold-start sequential recommendation. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM’22). 3421–3430.
[35]
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In Proceedings of the 5th International Conference on Learning Representations (ICLR’17): Workshop Track Proceedings. https://openreview.net/forum?id=HyhbYrGYe.
[36]
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. 2019. Adversarial robustness through local linearization. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS’19). 13824–13833.
[37]
Ruihong Qiu, Zi Huang, Tong Chen, and Hongzhi Yin. 2022. Exploiting positional information for session-based recommendation. ACM Transactions on Information Systems 40, 2 (2022), Article 35, 24 pages.
[38]
Ruihong Qiu, Zi Huang, and Hongzhi Yin. 2021. Memory augmented multi-instance contrastive predictive coding for sequential recommendation. In Proceedings of the IEEE International Conference on Data Mining (ICDM’21). IEEE, Los Alamitos, CA, 519–528.
[39]
Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. 2022. Contrastive learning for representation degeneration problem in sequential recommendation. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining (WSDM’22). ACM, New York, NY, 813–823.
[40]
Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017. Personalizing session-based recommendations with hierarchical recurrent neural networks. In Proceedings of the 11th ACM Conference on Recommender Systems (RecSys’17). ACM, New York, NY, 130–137.
[41]
Steffen Rendle. 2010. Factorization machines. In Proceedings of the 10th IEEE International Conference on Data Mining (ICDM’10). 995–1000.
[42]
Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010. Factorizing personalized Markov chains for next-basket recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW’10)ACM, New York, NY, 811–820.
[43]
Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019. BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM’19). ACM, New York, NY, 1441–1450.
[44]
Ke Sun, Yunbo Cao, Xinying Song, Young-In Song, Xiaolong Wang, and Chin-Yew Lin. 2009. Learning to recommend questions based on user ratings. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09). ACM, New York, NY, 751–758.
[45]
Rui Sun, Xuezhi Cao, Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang, and Kai Zheng. 2020. Multi-modal knowledge graphs for recommender systems. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM’20). 1405–1414.
[46]
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818–2826.
[47]
Jiaxi Tang and Ke Wang. 2018. Personalized top-N sequential recommendation via convolutional sequence embedding. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining (WSDM’18). ACM, New York, NY, 565–573.
[48]
Hongwei Wang, Fuzheng Zhang, Mengdi Zhang, Jure Leskovec, Miao Zhao, Wenjie Li, and Zhongyuan Wang. 2019. Knowledge-aware graph neural networks with label smoothness regularization for recommender systems. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’19). ACM, New York, NY, 968–977.
[49]
Jiachun Wang, Fajie Yuan, Jian Chen, Qingyao Wu, Min Yang, Yang Sun, and Guoxiao Zhang. 2021. StackRec: Efficient training of very deep sequential recommender models by iterative stacking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’21). 357–366.
[50]
Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019. Session-based recommendation with graph neural networks. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI’19). 346–353.
[51]
Xing Wu, Yibing Liu, Xiangyang Zhou, and Dianhai Yu. 2020. Distilling knowledge from pre-trained language models via text smoothing. arXiv preprint arXiv:2005.03848 (2020).
[52]
Xu Xie, Fei Sun, Zhaoyang Liu, Jinyang Gao, Bolin Ding, and Bin Cui. 2020. Contrastive pre-training for sequential recommendation. CoRR abs/2010.14395 (2020).
[53]
Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. Jose, and Xiangnan He. 2019. A simple convolutional generative network for next item recommendation. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM’19). ACM, New York, NY, 582–590.
[54]
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys 52, 1 (2019), 1–38.
[55]
Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, and Xiaofang Zhou. 2019. Feature-level deeper self-attention network for sequential recommendation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI’19). 4320–4326.
[56]
Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal intervention for leveraging popularity bias in recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’21). ACM, New York, NY, 11–20.
[57]
Wayne Xin Zhao, Yupeng Hou, Xingyu Pan, Chen Yang, Zeyu Zhang, Zihan Lin, Jingsen Zhang, et al. 2022. RecBole 2.0: Towards a more up-to-date recommendation library. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management. 4722–4726.
[58]
Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, et al. 2021. RecBole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management. 4653–4664.
[59]
Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020. S3-Rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management. 1893–1902.
[60]
Kun Zhou, Xiaolei Wang, Yuanhang Zhou, Chenzhan Shang, Yuan Cheng, Wayne Xin Zhao, Yaliang Li, and Ji-Rong Wen. 2021. CRSLab: An open-source toolkit for building conversational recommender system. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. 185–193.
[61]
Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Filter-enhanced MLP is all you need for sequential recommendation. In Proceedings of the ACM Web Conference 2022 (WWW’22). 2388–2399.
[62]
Kun Zhou, Wayne Xin Zhao, Shuqing Bian, Yuanhang Zhou, Ji-Rong Wen, and Jingsong Yu. 2020. Improving conversational recommender systems via knowledge graph based semantic fusion. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’20). 1006–1014.
[63]
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020. Towards topic-guided conversational recommender system. In Proceedings of the 28th International Conference on Computational Linguistics. 4128–4139.
[64]
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. FreeLB: Enhanced adversarial training for natural language understanding. In Proceedings of the Conference on Learning Representations (ICLR’20).

Cited By

View all

Index Terms

  1. Enhancing Multi-View Smoothness for Sequential Recommendation Models

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Information Systems
    ACM Transactions on Information Systems  Volume 41, Issue 4
    October 2023
    958 pages
    ISSN:1046-8188
    EISSN:1558-2868
    DOI:10.1145/3587261
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 April 2023
    Online AM: 31 January 2023
    Accepted: 05 January 2023
    Revised: 26 October 2022
    Received: 30 April 2022
    Published in TOIS Volume 41, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Sequential recommendation
    2. data smoothness
    3. model smoothness

    Qualifiers

    • Research-article

    Funding Sources

    • National Natural Science Foundation of China
    • Beijing Natural Science Foundation
    • Beijing Outstanding Young Scientist Program
    • Outstanding Innovative Talents Cultivation Funded Programs

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)356
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 15 Sep 2024

    Other Metrics

    Citations

    Cited By

    View all

    View Options

    Get Access

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    Full Text

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media