LLM vs small model? large language model based text augmentation enhanced personality detection model
AUTHORs: 
Linmei Hu, Hongyu He, Duokang Wang, Ziwang Zhao, Yingxia Shao, Liqiang NieAuthors Info & Claims
Linmei Hu, Hongyu He, Duokang Wang, Ziwang Zhao, Yingxia Shao, Liqiang NieAuthors Info & ClaimsArticle No.: 2034, Pages 18234 - 18242
Abstract
Personality detection aims to detect one's personality traits underlying in social media posts. One challenge of this task is the scarcity of ground-truth personality traits which are collected from self-report questionnaires. Most existing methods learn post features directly by fine-tuning the pre-trained language models under the supervision of limited personality labels. This leads to inferior quality of post features and consequently affects the performance. In addition, they treat personality traits as one-hot classification labels, overlooking the semantic information within them. In this paper, we propose a large language model (LLM) based text augmentation enhanced personality detection model, which distills the LLM's knowledge to enhance the small model for personality detection, even when the LLM fails in this task. Specifically, we enable LLM to generate post analyses (augmentations) from the aspects of semantic, sentiment, and linguistic, which are critical for personality detection. By using contrastive learning to pull them together in the embedding space, the post encoder can better capture the psycho-linguistic information within the post representations, thus improving personality detection. Furthermore, we utilize the LLM to enrich the information of personality labels for enhancing the detection performance. Experimental results on the benchmark datasets demonstrate that our model outperforms the state-of-the-art methods on personality detection.
References
[1]
Anusic, I.; Schimmack, U.; Pinkus, R.; and Lockwood, P. 2009. The Nature and Structure of Correlations Among Big Five Ratings: The Halo-Alpha-Beta Model. Journal of personality and social psychology, 97: 1142-56.
[2]
Bagby, R. M.; Gralnick, T. M.; Al-Dajani, N.; and Uliaszek, A. A. 2016. The Role of the Five-Factor Model in Personality Assessment and Treatment Planning. Clinical Psychology: Science and Practice, 23(4): 365-381.
[3]
Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 632-642.
[4]
Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.; Sutskever, I.; and Amodei, D. 2020. Language Models are Few-Shot Learners. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 1877-1901.
[5]
Cheng, Q.; Yang, X.; Sun, T.; Li, L.; and Qiu, X. 2023a. Improving Contrastive Learning of Sentence Embeddings from AI Feedback. In Findings of the Association for Computational Linguistics: ACL 2023, 11122-11138.
[6]
Cheng, Q.; Yang, X.; Sun, T.; Li, L.; and Qiu, X. 2023b. Improving Contrastive Learning of Sentence Embeddings from AI Feedback. In Findings of the Association for Computational Linguistics: ACL 2023, 11122-11138.
[7]
Cui, B.; and Qi, C. 2017. Survey Analysis of Machine Learning Methods for Natural Language Processing for MBTI Personality Type Prediction. Technical report, Stanford University.
[8]
Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q.; and Salakhutdinov, R. 2019. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Korhonen, A.; Traum, D.; and Marquez, L., eds., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2978-2988.
[9]
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171-4186.
[10]
Fang, Q.; Giachanou, A.; Bagheri, A.; Boeschoten, L.; van Kesteren, E.-J.; Shafiee Kamalabad, M.; and Oberski, D. 2023. On Text-based Personality Computing: Challenges and Future Directions. In Findings of the Association for Computational Linguistics: ACL 2023, 10861-10879.
[11]
Gao, T.; Yao, X.; and Chen, D. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6894-6910.
[12]
Gjurković, M.; Karan, M.; Vukojević, I.; Bošnjak, M.; and Snajder, J. 2021. PANDORA Talks: Personality and Demographics on Reddit. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, 138-152.
[13]
Hadsell, R.; Chopra, S.; and LeCun, Y. 2006. Dimensionality Reduction by Learning an Invariant Mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, 1735-1742.
[14]
Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531.
[15]
Hsieh, C.-Y.; Li, C.-L.; Yeh, C.-k.; Nakhost, H.; Fujii, Y.; Ratner, A.; Krishna, R.; Lee, C.-Y.; and Pfister, T. 2023. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes. In Findings of the Association for Computational Linguistics: ACL 2023, 8003-8017.
[16]
Iliopoulos, F.; Kontonis, V.; Baykal, C.; Menghani, G.; Trinh, K.; and Vee, E. 2022. Weighted Distillation with Unlabeled Examples. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 7024-7037.
[17]
Ji, Y.; Wu, W.; Zheng, H.; Hu, Y.; Chen, X.; and He, L. 2023. Is ChatGPT a Good Personality Recognizer? A Preliminary Study. arXiv:2307.03952.
[18]
Jiang, H.; Zhang, X.; and Choi, J. D. 2019. Automatic Text-based Personality Recognition on Monologues and Multiparty Dialogues Using Attentive Networks and Contextual Embeddings. arXiv:1911.09304.
[19]
Jiang, H.; Zhang, X.; and Choi, J. D. 2020. Automatic Text-Based Personality Recognition on Monologues and Multiparty Dialogues Using Attentive Networks and Contextual Embeddings (Student Abstract). volume 34, 13821-13822.
[20]
Johnson, R.; Wootten, M.; Spear, A.; and Smolensky, A. 2023. The Relationship Between Personality Traits and the Processing of Emotion Words: Evidence from Eye-Movements in Sentence Reading. Journal of Psycholinguistic Research, 1-27.
[21]
Keh, S. S.; and Cheng, I.-T. 2019. Myers-Briggs Personality Classification and Personality-Specific Language Generation Using Pre-trained Language Models. arXiv:1907.06333.
[22]
Kishima, R.; Matsumoto, K.; Yoshida, M.; and Kita, K. 2021. Construction of MBTI Personality Estimation Model Considering Emotional Information. In Proceedings of the 35th Pacific Asia Conference on Language, Information and Computation, 262-269. Shanghai, China.
[23]
Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692.
[24]
Lynn, V.; Balasubramanian, N.; and Schwartz, H. A. 2020. Hierarchical Modeling for User Personality Prediction: The Role of Message-Level Attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5306-5316.
[25]
Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P. F.; Leike, J.; and Lowe, R. 2022. Training language models to follow instructions with human feedback. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems, volume 35, 27730-27744.
[26]
Schwartz, H. A.; Eichstaedt, J. C.; Kern, M. L.; Dziurzynski, L.; Ramones, S. M.; Agrawal, M.; Shah, A.; Kosinski, M.; Stillwell, D.; Seligman, M. E. P.; and Ungar, L. H. 2013. Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach. PLOS ONE, 8(9): 1-16.
[27]
Stajner, S.; and Yenikent, S. 2020. A Survey of Automatic Personality Detection from Texts. In Proceedings of the 28th International Conference on Computational Linguistics, 6284-6295.
[28]
Sun, Y.; Wang, S.; Feng, S.; Ding, S.; Pang, C.; Shang, J.; Liu, J.; Chen, X.; Zhao, Y.; Lu, Y.; Liu, W.; Wu, Z.; Gong, W.; Liang, J.; Shang, Z.; Sun, P.; Liu, W.; Ouyang, X.; Yu, D.; Tian, H.; Wu, H.; and Wang, H. 2021. ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation. arXiv:2107.02137.
[29]
Tadesse, M. M.; Lin, H.; Xu, B.; and Yang, L. 2018. Personality Predictions Based on User Behavior on the Facebook Social Media Platform. IEEE Access, 6: 61959-61969.
[30]
Tandera, T.; Hendro; Suhartono, D.; Wongso, R.; and Prasetio, Y. L. 2017. Personality Prediction System from Facebook Users. Procedia Computer Science, 116: 604-611.
[31]
Discovery and innovation of computer science technology in artificial intelligence era: The 2nd International Conference on Computer Science and Computational Intelligence (ICCSCI 2017).
[32]
Tausczik, Y. R.; and Pennebaker, J. W. 2010. The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology, 29(1): 24-54.
[33]
Wang, S.; Liu, Y.; Xu, Y.; Zhu, C.; and Zeng, M. 2021. Want To Reduce Labeling Cost? GPT-3 Can Help. In Findings of the Association for Computational Linguistics: EMNLP 2021, 4195-4205.
[34]
Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2023a. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 13484-13508.
[35]
Wang, Z.; Xie, Q.; Ding, Z.; Feng, Y.; and Xia, R. 2023b. Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study. arXiv:2304.04339.
[36]
Wen, Z.; Cao, J.; Yang, R.; Liu, S.; and Shen, J. 2021. Automatically Select Emotion for Response via Personality-affected Emotion Transition. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 5010-5020.
[37]
Williams, A.; Nangia, N.; and Bowman, S. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1112-1122.
[38]
Xue, D.; Wu, L.; Hong, Z.; Guo, S.; Gao, L.; Wu, Z.; Zhong, X.; and Sun, J. 2018. Deep learning-based personality recognition from text posts of online social networks. Applied Intelligence, 48.
[39]
Yan, Y.; Li, R.; Wang, S.; Zhang, F.; Wu, W.; and Xu, W. 2021. ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 5065-5075.
[40]
Yang, F.; Quan, X.; Yang, Y.; and Yu, J. 2021a. Multi-Document Transformer for Personality Detection. volume 35, 14221-14229.
[41]
Yang, R.; Chen, J.; and Narasimhan, K. 2021. Improving Dialog Systems for Negotiation with Personality Modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 681-693.
[42]
Yang, T.; Deng, J.; Quan, X.; and Wang, Q. 2023. Orders Are Unwanted: Dynamic Deep Graph Convolutional Network for Personality Detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 13896-13904.
[43]
Yang, T.; Yang, F.; Ouyang, H.; and Quan, X. 2021b. Psycholinguistic Tripartite Graph Network for Personality Detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 4229-4239.
[44]
Yarkoni, T. 2010. Personality in 100,000 Words: A large-scale analysis of personality and word use among bloggers. Journal of Research in Personality, 44(3): 363-373.
[45]
Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022. OPT: Open Pre-trained Transformer Language Models. arXiv:2205.01068.
[46]
Zhang, T.; Ladhak, F.; Durmus, E.; Liang, P.; McKeown, K.; and Hashimoto, T. B. 2023. Benchmarking Large Language Models for News Summarization. arXiv:2301.13848.
[47]
Zhang, Y.; Jin, R.; and Zhou, Z.-H. 2010. Understanding bag-of-words model: a statistical framework. International Journal of Machine Learning and Cybernetics, 1: 43-52.
[48]
Zhu, Y.; Hu, L.; Ge, X.; Peng, W.; and Wu, B. 2022. Contrastive Graph Transformer Network for Personality Detection. In Raedt, L. D., ed., Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, 4559-4565.
Index Terms
- LLM vs small model? large language model based text augmentation enhanced personality detection model
Index terms have been assigned to the content through auto-classification.
Recommendations
Model for Personality Detection Based on Text Analysis
Advances in Computational IntelligenceAbstractPersonality is a unique trait which distinguish people from each other. It is a set of individual differences in thinking, feeling and behaving of people, and it affects interaction, relationships and environment of people. Personality can be ...
Towards a Model for Personality-Based Agents for Emotional Responses
Webmedia '16: Proceedings of the 22nd Brazilian Symposium on Multimedia and the WebAffective Computing is a promising research area with many open challenges. This area expects to develop computational systems that can monitor and respond to the affective states of an interacting user (IU). These affective states can be observed in ...
The Impact of Virtual Agent Personality on a Shared Mental Model with Humans during Collaboration
AAMAS '15: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent SystemsThe development of a Shared Mental Model (SMM) between team members and effective communication of the shared knowledge have been found to improve teamwork performance. However, humans may differ in how they produce and perceive communication acts ...
Comments
Information & Contributors
Information
Published In
February 2024
23861 pages
ISBN:978-1-57735-887-9
Copyright © 2024 Association for the Advancement of Artificial Intelligence.
Sponsors
- Association for the Advancement of Artificial Intelligence
Publisher
AAAI Press
Publication History
Published: 07 January 2025
Qualifiers
- Research-article
- Research
- Refereed limited
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 17 Jan 2025