Question Difficulty Consistent Knowledge Tracing
Pages 4239 - 4248
Abstract
Knowledge tracing aims to estimate knowledge states of students over a set of skills based on students' past learning activities. Deep learning based knowledge tracing models show superior performance to traditional knowledge tracing approaches. Early works like DKT use skill IDs and student responses only. Recent works also incorporate questions IDs into their models and achieve much improved performance in the next question correctness prediction task. However, predictions made by these models are thus on specific questions, and it is not straightforward to translate them to estimation of students' knowledge states over skills. In this paper, we propose to replace question IDs with question difficulty levels in deep knowledge tracing models. The predictions made by our model can be more readily translated to students' knowledge states over skills. Furthermore, by using question difficulty levels to replace question IDs, we can also alleviate the cold-start problem in knowledge tracing as online learning platforms are updated frequently with new questions. We further use two techniques to smooth the predicted scores. One is to combine embeddings of nearby difficulty levels using the Hann function. The other is to constrain the predicted probabilities to be consistent with question difficulties by imposing a penalty if they are not consistent. We conduct extensive experiments to study the performance of the proposed model. Our experimental results show that our model outperforms the state-of-the-art knowledge tracing models in terms of both accuracy and consistency with question difficulty levels.
Supplemental Material
MP4 File
Supplemental video
- Download
- 4.96 MB
References
[1]
Ghodai Abdelrahman and Qing Wang. 2019. Knowledge Tracing with Sequential Key-Value Memory Networks. In Proceedings of the 42nd International ACM Conference on Research and Development in Information Retrieval (SIGIR'19). ACM, New York, NY, USA, Paris, France, 175--184. https://doi.org/10.1145/3331184.3331195
[2]
Ghodai Abdelrahman, Qing Wang, and Bernardo Pereira Nunes. 2023. Knowledge Tracing: A Survey. Comput. Surveys, Vol. 55, 11 (2023), 1--37. https://doi.org/10.1145/3569576
[3]
Hao Cen, Kenneth R. Koedinger, and Brian Junker. 2006. Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement. In Proceedings of the 8th International Conference on Intelligent Tutoring Systems (ITS'06) (Lecture Notes in Computer Science, Vol. 4053). Springer, Jhongli, Taiwan, 164--175. https://doi.org/10.1007/11774303_17
[4]
Hao Cen, Kenneth R. Koedinger, and Brian Junker. 2008. Comparing Two IRT Models for Conjunctive Skills. In Proceedings of the 9th International Conference Intelligent Tutoring Systems (ITS'08) (Lecture Notes in Computer Science, Vol. 5091). Springer, Montreal, Canada, 796--798. https://doi.org/10.1007/978--3--540--69132--7_111
[5]
Jiahao Chen, Zitao Liu, Shuyan Huang, Qiongqiong Liu, and Weiqi Luo. 2023. Improving Interpretability of Deep Sequential Knowledge Tracing Models with Question-centric Cognitive Representations. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI'23). AAAI Press, Washington, DC, USA, 14196--14204. https://doi.org/10.1609/AAAI.V37I12.26661
[6]
Penghe Chen, Yu Lu, Vincent W. Zheng, and Yang Pian. 2018. Prerequisite-Driven Deep Knowledge Tracing. In Proceedings of the IEEE International Conference on Data Mining (ICDM'18). IEEE Computer Society, Singapore, 39--48. https://doi.org/10.1109/ICDM.2018.00019
[7]
Beno^i t Choffin, Fabrice Popineau, Yolaine Bourda, and Jill-Jê nn Vie. 2019. DAS3H: Modeling Student Learning and Forgetting for Optimally Scheduling Distributed Practice of Skills. In Proceedings of the 12th International Conference on Educational Data Mining (EDM'19). International Educational Data Mining Society (IEDMS), Montré al, Canada, 29--38.
[8]
Youngduck Choi, Youngnam Lee, Junghyun Cho, Jineon Baek, Byungsoo Kim, Yeongmin Cha, Dongmin Shin, Chan Bae, and Jaewe Heo. 2020. Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing. In Proceedings of the 7th ACM Conference on Learning at Scale. ACM, New York, NY, USA, USA, 341--344. https://doi.org/10.1145/3386527.3405945
[9]
Albert T. Corbett and John R. Anderson. 1994. Knowledge Tracing: Modelling the Acquisition of Procedural Knowledge. User Modeling and User-Adapted Interaction, Vol. 4, 4 (1994), 253--278. https://doi.org/10.1007/BF01099821
[10]
Ryan Shaun Joazeiro de Baker, Albert T. Corbett, and Vincent Aleven. 2008. More Accurate Student Modeling through Contextual Estimation of Slip and Guess Probabilities in Bayesian Knowledge Tracing. In Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS'08) (Lecture Notes in Computer Science). Springer, Montreal, Canada, 406--415. https://doi.org/10.1007/978--3--540--69132--7_44
[11]
Aritra Ghosh, Neil T. Heffernan, and Andrew S. Lan. 2020. Context-Aware Attentive Knowledge Tracing. In The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD'20). ACM, New York, NY, USA, CA, USA, 2330--2339. https://doi.org/10.1145/3394486.3403282
[12]
Shuyan Huang, Zitao Liu, Xiangyu Zhao, Weiqi Luo, and Jian Weng. 2023. Towards Robust Knowledge Tracing Models via k-Sparse Attention. In Proceedings of the 46th International ACM Conference on Research and Development in Information Retrieval (SIGIR'23). ACM, New York, NY, US, Taipei, Taiwan, 2441--2445. https://doi.org/10.1145/3539618.3592073
[13]
Tao Huang, Mengyi Liang, Huali Yang, Zhi Li, Tao Yu, and Shengze Hu. 2021. Context-aware knowledge tracing integrated with the exercise representation and association in mathematics. In Proceedings of the 14th International Conference on Educational Data Mining (EDM'21). International Educational Data Mining Society, virtual, 360--366.
[14]
Tanja K"a ser, Severin Klingler, Alexander G. Schwing, and Markus H. Gross. 2017. Dynamic Bayesian Networks for Student Modeling. IEEE Transactions on Learning Technologies, Vol. 10, 4 (2017), 450--462.
[15]
Mohammad Khajah, Robert V. Lindsey, and Michael Mozer. 2016. How Deep is Knowledge Tracing?. In Proceedings of the 9th International Conference on Educational Data Mining (EDM'16). International Educational Data Mining Society (IEDMS), Raleigh, North Carolina, USA, 94--101.
[16]
Andrew S. Lan, Christoph Studer, and Richard G. Baraniuk. 2014a. Time-varying learning and content analytics via sparse factor analysis. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'14). ACM, New York, NY, USA, New York, NY, USA, 452--461. https://doi.org/10.1145/2623330.2623631
[17]
Andrew S. Lan, Andrew E. Waters, Christoph Studer, and Richard G. Baraniuk. 2014b. Sparse factor analysis for learning and content analytics. Journal of Machine Learning Research, Vol. 15, 1 (2014), 1959--2008.
[18]
Wonsung Lee, Jaeyoon Chun, Youngmin Lee, Kyoungsoo Park, and Sungrae Park. 2022. Contrastive Learning for Knowledge Tracing. In The ACM Web Conference (WWW'22). ACM, New York, NY, US, Lyon, France, 2330--2338. https://doi.org/10.1145/3485447.3512105
[19]
RV Lindsey, JD Shroyer, H Pashler, and MC. Mozer. 2014. Improving students' long-term knowledge retention through personalized review. Psychological Science, Vol. 25, 3 (2014), 639--647.
[20]
Qi Liu, Zhenya Huang, Yu Yin, Enhong Chen, Hui Xiong, Yu Su, and Guoping Hu. 2021a. EKT: Exercise-Aware Knowledge Tracing for Student Performance Prediction. IEEE Transactions on Knowledge and Data Engineering, Vol. 33, 1 (2021), 100--115.
[21]
Qi Liu, Shuanghong Shen, Zhenya Huang, Enhong Chen, and Yonghe Zheng. 2021b. A Survey of Knowledge Tracing. CoRR, Vol. abs/2105.15106 (2021).
[22]
Yunfei Liu, Yang Yang, Xianyu Chen, Jian Shen, Haifeng Zhang, and Yong Yu. 2020. Improving Knowledge Tracing via Pre-training Question Embeddings. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI'20). ijcai.org, Virtual, 1577--1583. https://doi.org/10.24963/IJCAI.2020/219
[23]
Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Boyu Gao, Weiqi Luo, and Jian Weng. 2023. Enhancing Deep Knowledge Tracing with Auxiliary Tasks. In Proceedings of the ACM Web Conference (WWW'23). ACM, New York, NY, US, Austin, TX, USA, 4178--4187. https://doi.org/10.1145/3543507.3583866
[24]
Zitao Liu, Qiongqiong Liu, Jiahao Chen, Shuyan Huang, Jiliang Tang, and Weiqi Luo. 2022. pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models. In Annual Conference on Neural Information Processing Systems 2022 (NIPS'22). Neural Information Processing Systems Foundation, Inc. (NeurIPS), New Orleans, LA, USA, 18542--18555.
[25]
Sein Minn, Jill-Jê nn Vie, Koh Takeuchi, Hisashi Kashima, and Feida Zhu. 2022. Interpretable Knowledge Tracing: Simple and Efficient Student Modeling with Causal Relations. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI'22). AAAI Press, Virtual, 12810--12818. https://doi.org/10.1609/AAAI.V36I11.21560
[26]
Sein Minn, Yi Yu, Michel C. Desmarais, Feida Zhu, and Jill-Jê nn Vie. 2018. Deep Knowledge Tracing and Dynamic Student Classification for Knowledge Tracing. In IEEE International Conference on Data Mining (ICDM'18). IEEE Computer Society, Singapore, 1182--1187. https://doi.org/10.1109/ICDM.2018.00156
[27]
Koki Nagatani, Qian Zhang, Masahiro Sato, Yan-Ying Chen, Francine Chen, and Tomoko Ohkuma. 2019. Augmenting Knowledge Tracing by Considering Forgetting Behavior. In The World Wide Web Conference (WWW'19). ACM, New York, NY, USA, San Francisco, CA, USA, 3101--3107. https://doi.org/10.1145/3308558.3313565
[28]
Hiromi Nakagawa, Yusuke Iwasawa, and Yutaka Matsuo. 2019. Graph-based Knowledge Tracing: Modeling Student Proficiency Using Graph Neural Network. In Proceedings of 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI'19). ACM, New York, NY, USA, Thessaloniki, Greece, 156--163. https://doi.org/10.1145/3350546.3352513
[29]
Shalini Pandey and George Karypis. 2019. A Self Attentive model for Knowledge Tracing. In Proceedings of the 12th International Conference on Educational Data Mining (EDM'19). International Educational Data Mining Society (IEDMS), Montré al, Canada, 384--389.
[30]
Shalini Pandey and Jaideep Srivastava. 2020. RKT: Relation-Aware Self-Attention for Knowledge Tracing. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM'20). ACM, New York, NY, USA, Ireland, 1205--1214. https://doi.org/10.1145/3340531.3411994
[31]
Zachary A. Pardos and Neil T. Heffernan. 2010. Modeling Individualization in a Bayesian Networks Implementation of Knowledge Tracing. In Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization (UMAP'10). Springer, Big Island, HI, USA, 255--266. https://doi.org/10.1007/978--3--642--13470--8_24
[32]
Zachary A. Pardos and Neil T. Heffernan. 2011. KT-IDEM: Introducing Item Difficulty to the Knowledge Tracing Model. In Proceedings of the 19th International Conference on User Modeling, Adaption and Personalization (UMAP'11). Springer, Girona, Spain, 243--254. https://doi.org/10.1007/978--3--642--22362--4_21
[33]
Philip I. Pavlik, Hao Cen, and Kenneth R. Koedinger. 2009. Performance Factors Analysis - A New Alternative to Knowledge Tracing. In Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED'09). IOS Press, Brighton, UK, 531--538.
[34]
Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J. Guibas, and Jascha Sohl-Dickstein. 2015. Deep Knowledge Tracing. In Annual Conference on Neural Information Processing Systems (NIPS'15). MIT Press, Cambridge, MA, Montreal, Quebec, Canada, 505--513.
[35]
Shuanghong Shen, Zhenya Huang, Qi Liu, Yu Su, Shijin Wang, and Enhong Chen. 2022. Assessing Student's Dynamic Knowledge State by Exploring the Question Difficulty Effect. In Proceedings of the 45th International ACM Conference on Research and Development in Information Retrieval (SIGIR'22). ACM, New York, NY, US, Madrid, Spain, 427--437. https://doi.org/10.1145/3477495.3531939
[36]
Shuanghong Shen, Qi Liu, Enhong Chen, Zhenya Huang, Wei Huang, Yu Yin, Yu Su, and Shijin Wang. 2021. Learning Process-consistent Knowledge Tracing. In The 27th ACM Conference on Knowledge Discovery and Data Mining (KDD'21). ACM, New York, NY, US, Singapore, 1452--1460. https://doi.org/10.1145/3447548.3467237
[37]
Shuanghong Shen, Qi Liu, Enhong Chen, Han Wu, Zhenya Huang, Weihao Zhao, Yu Su, Haiping Ma, and Shijin Wang. 2020. Convolutional Knowledge Tracing: Modeling Individualization in Student Learning Process. In Proceedings of the 43rd International ACM conference on research and development in Information Retrieval (SIGIR'20). ACM, New York, NY, USA, China, 1857--1860. https://doi.org/10.1145/3397271.3401288
[38]
Dongmin Shin, Yugeun Shim, Hangyeol Yu, Seewoo Lee, Byungsoo Kim, and Youngduck Choi. 2021. SAINT: Integrating Temporal Features for EdNet Correctness Prediction. In Proceedings of the 11th International Learning Analytics and Knowledge Conference (LAK'21). ACM, New York, NY, USA, Irvine, CA, USA, 490--496. https://doi.org/10.1145/3448139.3448188
[39]
J. Stamper, A. Niculescu-Mizil, S. Ritter, G.J. Gordon, and K.R. Koedinger. 2010. [Challenge/Development] data set from KDD Cup 2010 Educational Data Mining Challenge. https://doi.org/10.1145/3477495.3531939
[40]
Shiwei Tong, Qi Liu, Wei Huang, Zhenya Huang, Enhong Chen, Chuanren Liu, Haiping Ma, and Shijin Wang. 2020. Structure-based Knowledge Tracing: An Influence Propagation View. In Proceedings of the 20th IEEE International Conference on Data Mining (ICDM'20). IEEE, Sorrento, Italy, 541--550. https://doi.org/10.1109/ICDM50108.2020.00063
[41]
Duc Kinh Le Tran. 2021. Riiid! Answer Correctness Prediction Kaggle Challenge 4th Place Solution Summary. https://drive.google.com/file/d/1n9UQEuCk7YjPPqJ4ihcQPz6bueJqKKAf/view
[42]
Jill-Jê nn Vie and Hisashi Kashima. 2019. Knowledge Tracing Machines: Factorization Machines for Knowledge Tracing. In The 33rd AAAI Conference on Artificial Intelligence (AAAI'19). AAAI Press, Honolulu, Hawaii, USA, 750--757. https://doi.org/10.1609/AAAI.V33I01.3301750
[43]
Tianqi Wang, Fenglong Ma, and Jing Gao. 2019b. Deep Hierarchical Knowledge Tracing. In Proceedings of the 12th International Conference on Educational Data Mining (EDM'19). International Educational Data Mining Society (IEDMS), Montré al, Canada, 667--670.
[44]
Xinping Wang, Liangyu Chen, and Min Zhang. 2023. Deep Attentive Model for Knowledge Tracing. In Proceedings of the 37th AAAI Conference on Artificial Intelligence (AAAI'23). AAAI Press, Washington, DC, USA, 10192--10199. https://doi.org/10.1609/AAAI.V37I8.26214
[45]
Zhiwei Wang, Xiaoqin Feng, Jiliang Tang, Gale Yan Huang, and Zitao Liu. 2019a. Deep Knowledge Tracing with Side Information. In Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED'19) (Lecture Notes in Computer Science, Vol. 11626). Springer, Chicago, IL, USA, 303--308. https://doi.org/10.1007/978--3-030--23207--8_56
[46]
Yang Yang, Jian Shen, Yanru Qu, Yunfei Liu, Kerong Wang, Yaoming Zhu, Weinan Zhang, and Yong Yu. 2020. GIKT: A Graph-Based Interaction Model for Knowledge Tracing. In Proceedings of European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD'20). Springer, Ghent, Belgium, 299--315. https://doi.org/10.1007/978--3-030--67658--2_18
[47]
Chun-Kit Yeung and Dit-Yan Yeung. 2018. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale. ACM, New York, NY, US, London, UK, 5:1--5:10. https://doi.org/10.1145/3231644.3231647
[48]
Yu Yin, Le Dai, Zhenya Huang, Shuanghong Shen, Fei Wang, Qi Liu, Enhong Chen, and Xin Li. 2023. Tracing Knowledge Instead of Patterns: Stable Knowledge Tracing with Diagnostic Transformer. In Proceedings of the ACM Web Conference (WWW'23). ACM, New York, NY, US, Austin, TX, USA, 855--864. https://doi.org/10.1145/3543507.3583255
[49]
Michael Yudelson, Kenneth R. Koedinger, and Geoffrey J. Gordon. 2013. Individualized Bayesian Knowledge Tracing Models. In Proceedings of the 16th International ConferenceArtificial Intelligence in Education (AIED'13). Springer, Memphis, TN, USA, 171--180. https://doi.org/10.1007/978--3--642--39112--5_18
[50]
Junrui Zhang, Yun Mo, Changzhi Chen, and Xiaofeng He. 2021. GKT-CD: Make Cognitive Diagnosis Model Enhanced by Graph-based Knowledge Tracing. In Proceedings of International Joint Conference on Neural Networks (IJCNN'21). IEEE, Shenzhen, China, 1--8. https://doi.org/10.1109/IJCNN52387.2021.9533298
[51]
Jiani Zhang, Xingjian Shi, Irwin King, and Dit-Yan Yeung. 2017. Dynamic Key-Value Memory Networks for Knowledge Tracing. In Proceedings of the 26th International Conference on World Wide Web (WWW'17). ACM, New York, NY, US, Perth, Australia, 765--774. https://doi.org/10.1145/3038912.3052580 io
Index Terms
- Question Difficulty Consistent Knowledge Tracing
Recommendations
What Does Time Tell? Tracing the Forgetting Curve Using Deep Knowledge Tracing
Artificial Intelligence in EducationAbstractRecurrent Neural Network (RNN) based Deep Knowledge Tracing (DKT) can extract a complex representation of student knowledge just using the historical time series of correct-incorrect responses given as input and can predict the student’s ...
EDKT: An Extensible Deep Knowledge Tracing Model for Multiple Learning Factors
Database Systems for Advanced ApplicationsAbstractKnowledge Tracing (KT) refers to the problem of predicting learners’ future potential performance given their past learning history in e-learning systems. In order to better trace the learners’ knowledge, KT tasks have become increasingly ...
Self-attention in Knowledge Tracing: Why It Works
Artificial Intelligence in EducationAbstractKnowledge tracing refers to the dynamic assessment of a learner’s mastery of skills. There has been widespread adoption of the self-attention mechanism in knowledge-tracing models in recent years. These models consistently report performance gains ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Ministry of Education (MOE) Singapore
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 641Total Downloads
- Downloads (Last 12 months)641
- Downloads (Last 6 weeks)137
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in