Transfer learning for semantic similarity measures based on symbolic regression
Pages 37 - 49
Abstract
Recently, transfer learning strategies have become ideal for reusing acquired knowledge through a training phase. The key idea is that reusing such knowledge brings advantages such as increased accuracy and considerable resource savings. In this work, we design a novel strategy for effective and efficient transfer learning in semantic similarity. Our approach is based on generating and transferring optimal models obtained through a symbolic regression process being able to stack evaluation scores from several fundamental techniques. After an exhaustive empirical study, the results lead to high accuracy in addition to significant savings in terms of training time consumed in most of the scenarios considered.
References
[1]
Adadi A. and Berrada M., Peeking inside the blackbox: A survey on explainable artificial intelligence (XAI), IEEE Access 6(2018), 52138–52160.
[2]
Affenzeller M., Winkler S.M., Kronberger G., Kommenda M., Burlacuand B. and Wagner S. Gaining deeper insights in symbolic regression. In Genetic Programming Theory and Practice XI [GPTP 2013, University of Michigan, Ann Arbor, USA, May 9-11, 2013]. (2013), pp. 175–190.
[3]
Afzal N., Wang Y. and Liu H., Mayonlp at semeval-2016 task 1: Semantic textual similarity based on lexical semantic net and deep learning semantic model. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval-NAACL-HLT 2016, San Diego, CA, USA, 733 June 16-17, 2016 (2016), pp. 674–679.
[4]
Bollegala D., Matsuo Y. and Ishizuka M., A web search engine-based approach to measure semantic similarity between words, IEEE Trans Knowl Data Eng 23(7) (2011), 977–990.
[5]
Chaves-Gonzalez J.M. and Martinez-Gil J., Evolutionary algorithm based on different semantic similarity functions for synonym recognition in the biomedical domain, Knowl.-Based Syst 37 (2013), 62–69.
[6]
Deerwester S.C., Dumais S.T., Landauer T.K., Furnas G.W. and Harshman R.A., Indexing by latent semantic analysis, J Am Soc InfSci 41(6) (1990), 391–407.
[7]
Devlin J., Chang M., Lee K. and Toutanova K., BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), (2019), pp. 4171–4186. Association for Computational Linguistics.
[8]
Dinh T.T.H., Chu T.H. and Nguyen Q.U., Transfer learning in genetic programming. In IEEE Congress on Evolutionary Computation, CEC 2015, Sendai, Japan, May 25-28, 2015 (2015), pp. 1145–1151. IEEE.
[9]
Elbaz K., Shen S.-L., Zhou A., Yin Z.-Y. and Lyu H.-M., Predictionof disc cutter life during shield tunneling with ai via theincorporation of a genetic algorithm into a gmdh-type neural network, Engineering 7(2) (2021), 238–251.
[10]
Elbaz K., Yan T., Zhou A. and Shen S.-L., Deep learning analysis forenergy consumption of shield tunneling machine drive system, Tunnelling and Underground Space Technology 123 (2022), 104405.
[11]
Finkelstein L., Gabrilovich E., Matias Y., Rivlin E., Solan Z., Wolfman G. and Ruppin E., Placing search in context: the concept revisited, ACM Trans Inf Syst 20(1) (2002), 116–131.
[12]
Han L., Kashyap A.L., Finin T., Mayfield J. and Weese J., Umbc ebiquity-core: Semantic textual similarity systems. In M.T. Diab, T. Baldwin, and M. Baroni, editors, Proceedings of the Second Joint Conference on Lexical and Computational Semantics, *SEM 2013, June 13-14, 2013, Atlanta, Georgia, USA, (2013), pages 44–52. Association for Computational Linguistics.
[13]
Haslam E., Xue B. and Zhang M., Further investigation on genetic programming with transfer learning for symbolic regression. In IEEE Congress on Evolutionary Computation, CEC 2016, Vancouver, BC, Canada, July 24-29, 2016 (2016), pp. 3598–3605. IEEE.
[14]
Hill F., Reichart R. and Korhonen A., Simlex-999: Evaluating semantic models with (genuine) similarity estimation, Comput Linguistics 41(4) (2015), 665–695.
[15]
Iqbal M., Xue B., Al-Sahaf H. and Zhang M., Crossdomain reuse ofextracted knowledge in genetic programming for image classification, IEEE Trans Evol Comput 21(4) (2017), 569–587.
[16]
Jiang J.J. and Conrath D.W., Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th Research on Computational Linguistics International Conference, ROCLING 1997, Taipei, Taiwan, August 1997 (1997), pp. 19–33.
[17]
Koza J.R., Genetic programming: on the programming of computers by means of natural selection, volume 1. MIT press. (1992).
[18]
Lastra-Diaz J.J., Garcia-Serrano A., Batet M., Fernandez M. and Chirigati F., HESML: A scalable ontology-based semantic similarity measures library with a set of reproducible experiments and a replication dataset, Inf Syst 66 (2017), 97–118.
[19]
Lastra-Diaz J.J., Goikoetxea J., Taieb M.A.H., Garcia-Serrano A., Aouicha M.B. and Agirre E., A reproducible survey on word embeddings and ontology-based methods for word similarity: Linear combination soutperform the state of the art, Eng Appl Artif Intell 85 (2019), 645–665.
[20]
Leacock C., Chodorow M. and Miller G.A., Using corpus statistics and wordnet relations for sense identification, Comput Linguistics 24(1) (1998), 147–165.
[21]
Li Y., Bandar Z. and McLean D., An approach for measuring semantic similarity between words using multiple information sources, IEEE Trans Knowl Data Eng 15(4) (2003), 871–882.
[22]
Lin D., An information-theoretic definition of similarity. In Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998 (1998), pp. 296–304.
[23]
Luke S. and Panait L., A comparison of bloat control methods forgenetic programming, Evol Comput 14(3) (2006), 309–344.
[24]
Lundberg S.M. and Lee S., A unified approach to interpreting model predictions. In I. Guyon, von Luxburg, U., S. Bengio, H.M. Wallach, R. Fergus, Vishwanathan, S. V. N., and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, (2017), pp. 4765–4774.
[25]
Martinez-Gil J., Semantic similarity aggregators for very shorttextual expressions: a case study on landmarks and points ofinterest, J Intell Inf Syst 53(2) (2019), 361–380.
[26]
Martinez-Gil J. and Chaves-Gonzalez J.M., Automatic design ofsemantic similarity controllers based on fuzzy logics, ExpertSyst Appl 131 (2019), 45–59.
[27]
Martinez-Gil J. and Chaves-Gonzalez J.M., A novel method based onsymbolic regression for interpretable semantic similaritymeasurement, Expert Syst Appl 160 (2020), 113663.
[28]
Martinez-Gil J. and Chaves-Gonzalez J.M., Semantic similarity controllers: On the trade-off between accuracy and interpretability, Knowledge-Based Systems (2021), pp. 107609.
[29]
Martinez-Gil J. and Chaves-Gonzalez J.M., Sustainable semanticsimilarity assessment, Journal of Intelligent & Fuzzy Systems 43(5) (2022), 6163–6174.
[30]
Mikolov T., Sutskever I., Chen K., Corrado G.S. and Dean J., Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26:27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States., (2013), pp. 3111–3119.
[31]
Miller G. and Charles W., Contextual correlates of semantic similarity, Language and Cognitive Processes 6(1) (1991), 1–28.
[32]
Munoz L., Trujillo L. and Silva S., Transfer learning inconstructive induction with genetic programming, Genet Program Evolvable Mach 21(4) (2020), 529–569.
[33]
O’Neill D., Al-Sahaf H., Xue B. and Zhang M., Common subtrees in related problems: A novel transfer learning approach for genetic programming. In 2017 IEEE Congress on Evolutionary Computation, CEC 2017, Donostia, San Sebastian, Spain, June 5-8, 2017, (2017), pp. 1287–1294. IEEE.
[34]
Pan S.J. and Yang Q., A survey on transfer learning, IEEE Trans Knowl Data Eng 22(10) (2010), 1345–1359.
[35]
Peters M.E., Neumann M., Iyyer M., Gardner M., Clark C., Lee K. and Zettlemoyer L., editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), (2018), pp. 2227–2237. Association for Computational Linguistics.
[36]
Rada R., Mili H., Bicknell E. and Blettner M., Development and application of a metric on semantic nets, IEEE Trans Syst ManCybern 19(1) (1989), 17–30.
[37]
Resnik P., Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language, J Artif Intell Res 11 (1999), 95–130.
[38]
Rubenstein H. and Goodenough J.B., Contextual correlates of synonymy, Communications of the ACM 8(10) (1965), 627–633.
[39]
Shen S.-L., Elbaz K., Shaban W.M. and Zhou A., Real-time prediction of shield moving trajectory during tunnelling, Acta Geotechnica 17(4) (2022), 1533–1549.
[40]
Su S., Li W., Mou J., Garg A., Gao L. and Liu J., A hybrid battery equivalent circuit model, deep learning, and transfer learning for battery state monitoring, IEEE Transactions on Transportation Electrification (2022).
[41]
Tan C., Sun F., Kong T., Zhang W., Yang C. and Liu C., A survey on deep transfer learning. In V. Kurkova, Y. Manolopoulos, B. Hammer, L.S. Iliadis, and I. Maglogiannis, editors, Artificial Neural Networks and Machine Learning - ICANN 2018 - 27th International Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III, volume 11141 of Lecture Notes in Computer Science, (2018), pp. 270–279. Springer.
[42]
Vladislavleva E., Smits G. and den Hertog D., On the importance ofdata balancing for symbolic regression, IEEE Trans Evolutionary Computation 14(2) (2010), 252–277.
[43]
Wu Z. and Palmer M.S., Verb semantics and lexical selection. In J. Pustejovsky, editor, 32nd Annual Meeting of the Association for Computational Linguistics, 27-30 June 1994, New Mexico State University, Las Cruces, New Mexico, USA, Proceedings, (1994), pp. 133–138. Morgan Kaufmann Publishers / ACL.
[44]
Zhu G. and Iglesias C.A., Computing semantic similarity of conceptsin knowledge graphs, IEEE Trans Knowl Data Eng 29(1) (2017), 72–85.
Index Terms
- Transfer learning for semantic similarity measures based on symbolic regression
Index terms have been assigned to the content through auto-classification.
Recommendations
Transfer learning using computational intelligence
Transfer learning aims to provide a framework to utilize previously-acquired knowledge to solve new but similar problems much more quickly and effectively. In contrast to classical machine learning methods, transfer learning methods exploit the ...
Hypothesis transfer learning based on structural model similarity
AbstractTransfer learning focuses on building better predictive models by exploiting knowledge gained in previous related tasks, being able to soften the traditional supervised learning assumption of having identical train–test distributions. Most efforts ...
Big data, lifelong machine learning and transfer learning
WSDM '13: Proceedings of the sixth ACM international conference on Web search and data miningA major challenge in today's world is the Big Data problem, which manifests itself in Web and Mobile domains as rapidly changing and heterogeneous data streams. A data-mining system must be able to cope with the influx of changing data in a continual ...
Comments
Information & Contributors
Information
Published In
© 2023 – IOS Press. All rights reserved.
Publisher
IOS Press
Netherlands
Publication History
Published: 01 January 2023
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 06 Jan 2025