research-article

Application of back-translation: a transfer learning approach to identify ambiguous software requirements

Authors:

Ishan Mani Subedi,

Maninder Singh,

Vijayalakshmi Ramasamy,

Gursimran Singh WaliaAuthors Info & Claims

ACMSE '21: Proceedings of the 2021 ACM Southeast Conference

Pages 130 - 137

https://doi.org/10.1145/3409334.3452068

Published: 10 May 2021 Publication History

Abstract

Ambiguous requirements are problematic in requirement engineering as various stakeholders can debate on the interpretation of the requirements leading to a variety of issues in the development stages. Since requirement specifications are usually written in natural language, analyzing ambiguous requirements is currently a manual process as it has not been fully automated to meet the industry standards. In this paper, we used transfer learning by using ULMFiT where we pre-trained our model to a general-domain corpus and then fine-tuned it to classify ambiguous vs unambiguous requirements (target task). We then compared its accuracy with machine learning classifiers like SVM, Linear Regression, and Multinomial Naive Bayes. We also used back translation (BT) as a text augmentation technique to see if it improved the classification accuracy. Our results showed that ULMFiT achieved higher accuracy than SVM (Support Vector Machines), Logistic Regression and Multinomial Naive Bayes for our initial data set. Further by augmenting requirements using BT, ULMFiT got a higher accuracy than SVM, Logistic Regression, and Multinomial Naive Bayes classifier, improving the initial performance by 5.371%. Our proposed research provides some promising insights on how transfer learning and text augmentation can be applied to small data sets in requirements engineering.

References

[1]

O. Abdelwahab, M. Bahgat, C. J. Lowrance, and A. Elmaghraby. 2015. Effect of Training Set Size on SVM and Naive Bayes for Twitter Sentiment Analysis. In 2015 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT). 46--51.

Digital Library

[2]

A. Adhikari, A. Ram, R. Tang, and J. Lin. 2019. Rethinking Complex Neural Network Architectures for Document Classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 4046--4051.

[3]

G. Antoniol, K. Ayari, M. D. Penta, F. Khomh, and Y. Guéhéneuc. 2008. Is It a Bug or an Enhancement? A Text-based Approach to Classify Change Requests. In Proceedings of the 2008 Conference of the Center for Advanced Studies on Collaborative Research: Meeting of Minds. 304--318.

[4]

V. K. Anu, W. Hu, J. C. Carver, G. S. Walia, and G. Bradshaw. 2018. Development of a Human Error Taxonomy for Software Requirements: A Systematic Literature Review. Inf. Softw. Technol. 103 (2018), 112--124.

[5]

S. Bird, E. Klein, and E. Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. "O'Reilly Media, Inc.".

[6]

A. Bosu, M. Greiler, and C. Bird. 2015. Characteristics of Useful Code Reviews: An Empirical Study at Microsoft. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 146--156.

[7]

D. Cer, Y. Yang, S. Kong, N. Hua, N. Limtiaco, R. St. John, N. Constant, M. Guajardo-Céspedes, S. Yuan, C. Tar, et al. 2018. Universal Sentence Encoder. arXiv preprint arXiv:1803.11175 (2018).

[8]

F. Chantree, B. Nuseibeh, A. D. Roeck, and A. Willis. 2006. Identifying Nocuous Ambiguities in Natural Language Requirements. In 14th IEEE International Requirements Engineering Conference (RE'06). 59--68.

Digital Library

[9]

F. Fabbrini, M. Fusani, S. Gnesi, and G. Lami. 2000. Quality Evaluation of Software Requirement Specifications. In Proceedings of the Software and Internet Quality Week 2000 Conference. 1--18.

[10]

A. Ferrari, B. Donati, and S. Gnesi. 2017. Detecting Domain-specific Ambiguities: An NLP Approach Based on Wikipedia Crawling and Word Embeddings. In 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW). IEEE Computer Society, Los Alamitos, CA, USA, 393--399.

[11]

A. Ferrari, G. Gori, B. Rosadini, I. Trotta, S Bacherini, A. Fantechi, and S. Gnesi. 2018. Detecting Requirements Defects with NLP Patterns: An Industrial Experience in the Railway Domain. Empirical Software Engineering 23, 6 (2018), 3684--3733.

Digital Library

[12]

B. Gleich, O. Creighton, and L. Kof. 2010. Ambiguity Detection: Towards a Tool Explaining Ambiguity Sources. In International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 218--232.

[13]

A. Goswami, G. Walia, and A. Singh. 2015. Using Learning Styles of Software Professionals to Improve their Inspection Team Performance. International Journal of Software Engineering and Knowledge Engineering 25, 09n10 (2015), 1721--1726.

[14]

A. Goswami and G. S. Walia. 2016. Teaching Software Requirements Inspections to Software Engineering Students through Practical Training and Reflection. The ASEE Computers in Education (CoED) Journal 7, 4 (2016), 2.

[15]

A. Goswami, G. S. Walia, M. McCourt, and G. Padmanabhan. 2016. Using Eye Tracking to Investigate Reading Patterns and Learning Styles of Software Requirement Inspectors to Enhance Inspection Team Outcome. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2016, Ciudad Real, Spain, September 8--9, 2016. ACM, 34:1--34:10.

Digital Library

[16]

A. Goswami, G. S. Walia, and U. Rathod. 2016. Using Learning Styles to Staff and Improve Software Inspection Team Performance. In 2016 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). 9--12. htttps://

[17]

J. Howard et al. 2018. Fastai. https://github.com/fastai/fastai.

[18]

J. Howard and S. Gugger. 2020. Deep Learning for Coders with Fastai and PyTorch. O'Reilly Media.

[19]

J. Howard and S. Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. arXiv preprint arXiv:1801.06146 (2018).

[20]

E. Kamsties, D.M. Berry, and B. Paech. 2001. Detecting Ambiguities in Requirements Documents using Inspections. In Proceedings of the First Workshop on Inspection in Software Engineering (WISE'01).

[21]

Google LLC. 2020. Understanding the BLEU Score. https://cloud.google.com/translate/automl/docs/evaluate.

[22]

S. Merity, N. S. Keskar, and R. Socher. 2017. Regularizing and Optimizing LSTM Language Models. arXiv preprint arXiv:1708.02182 (2017).

[23]

K. Papineni, S. Roukos, T. Ward, and W. Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 311--318.

Digital Library

[24]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[25]

H. A. Rowley, S. Baluja, and T. Kanade. 1998. Neural Network-based Face Detection. IEEE Trans. Pattern Analysis and Machine Intelligence 20, 1 (1998).

Digital Library

[26]

S. Ruder. 2017. Transfer Learning - Machine Learning's Next Frontier. http://ruder.io/transfer-learning/.

[27]

M. Sahami, S. Dumais, D. Heckerman, and E. Horvitz. 1998. A Bayesian Approach to Filtering Junk E-mail. In Learning for Text Categorization: Papers from the 1998 Workshop, Vol. 62. Madison, Wisconsin, 98--105.

[28]

R. Sennrich, B. Haddow, and A. Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 86--96.

[29]

S. Shleifer. 2019. Low Resource Text Classification with Ulmfit and Backtranslation. arXiv preprint arXiv:1903.09244 (2019).

[30]

M. Singh. 2018. Automated Validation of Requirement Reviews: A Machine Learning Approach. In 2018 IEEE 26th International Requirements Engineering Conference (RE). 460--465.

[31]

M. Singh, V. Anu, G. S. Walia, and A. Goswami. 2018. Validating Requirements Reviews by Introducing Fault-type Level Granularity: A Machine Learning Approach. In Proceedings of the 11th Innovations in Software Engineering Conference (Hyderabad, India) (ISEC '18). Association for Computing Machinery, New York, NY, USA, Article 10, 11 pages.

Digital Library

[32]

M. Singh, G. Walia, and A. Goswami. 2018. Using Supervised Learning to Guide the Selection of Software Inspectors in Industry. In 2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). IEEE Computer Society, Los Alamitos, CA, USA, 12--17.

[33]

M. Singh, G. S. Walia, and A. Goswami. 2017. Validation of Inspection Reviews over Variable Features Set Threshold. In 2017 International Conference on Machine Learning and Data Science (MLDS). 128--135.

[34]

L. N. Smith. 2018. A Disciplined Approach to Neural Network Hyper-parameters: Part 1-Learning Rate, Batch Size, Momentum, And Weight Decay. arXiv preprint arXiv:1803.09820 (2018).

[35]

G. Tassey. 2002. The Economic Impacts of Inadequate Infrastructure for Software Testing. National Institute of Standards and Technology.

[36]

S. F. Tjong and D. M. Berry. 2013. The Design of SREE --- A Prototype Potential Ambiguity Finder for Requirements Specifications and Lessons Learned. In Requirements Engineering: Foundation for Software Quality. Springer Berlin Heidelberg, Berlin, Heidelberg, 80--95.

[37]

B. V. Burgh and S. Verberne. 2019. The Merits of Universal Language Model Fine-tuning for Small Datasets-A Case with Dutch Book Reviews. arXiv preprint arXiv:1910.00896 (2019).

[38]

J. Wei and K. Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 6382--6388.

[39]

H. Yang, A. D. Roeck, V. Gervasi, A. Willis, and B. Nuseibeh. 2011. Analysing Anaphoric Ambiguity in Natural Language Requirements. Requirements Engineering 16, 3 (2011), 163.

Digital Library

Cited By

Pan RIbrahimzada AKrishna RSankar DWassi LMerler MSobolev BPavuluri RSinha SJabbarvand RRoychoudhury APaiva AAbreu RStorey M(2024)Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating CodeProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639226(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639226
Korfmann RBeyersdorffer PMünch JKuhrmann M(2024)Using Data Augmentation to Support AI-Based Requirements Evaluation in Large-Scale ProjectsSystems, Software and Services Process Improvement10.1007/978-3-031-71139-8_7(97-111)Online publication date: 7-Sep-2024
https://doi.org/10.1007/978-3-031-71139-8_7
Hadi MFard F(2023)Evaluating pre-trained models for user feedback analysis in software engineering: a study on classification of app-reviewsEmpirical Software Engineering10.1007/s10664-023-10314-x28:4Online publication date: 23-May-2023
https://dl.acm.org/doi/10.1007/s10664-023-10314-x
Show More Cited By

Index Terms

Application of back-translation: a transfer learning approach to identify ambiguous software requirements
1. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Requirements analysis

Recommendations

Using Machine Learning to Convey Emotions During Requirements Elicitation Interviews
ACMSE '19: Proceedings of the 2019 ACM Southeast Conference

In an effort to assist in an ongoing research project where stakeholders have been interviewed using voice recording platforms and the Empatica E4 wristband to gather biofeedback data, the purpose of this research project in relation to the ...
A Zero-Shot Learning Approach to Classifying Requirements: A Preliminary Study
Requirements Engineering: Foundation for Software Quality
Abstract
Context and motivation: Advances in Machine Learning (ML) and Deep Learning (DL) technologies have transformed the field of Natural Language Processing (NLP), making NLP more practical and accessible. Motivated by these exciting developments, ...
Research on transfer learning algorithm based on support vector machine
Fuzzy Systems for Medical Image Analysis

Transfer learning is a new machine learning algorithm. It solves problems in different but related target domains by utilizing the knowledge in existing data. Based on the classical SVM algorithm and transfer learning, a selective transfer learning ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ACMSE '21: Proceedings of the 2021 ACM Southeast Conference

April 2021

263 pages

ISBN:9781450380683

DOI:10.1145/3409334

Conference Chair:
Kazi Rahman
Jacksonville State University
,
Program Chair:
Eric Gamess
Jacksonville State University, Jacksonville, Alabama, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ACM SE '21

Sponsor:

ACM

ACM SE '21: 2021 ACM Southeast Conference

April 15 - 17, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 502 of 1,023 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
192
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)7

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan RIbrahimzada AKrishna RSankar DWassi LMerler MSobolev BPavuluri RSinha SJabbarvand RRoychoudhury APaiva AAbreu RStorey M(2024)Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating CodeProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639226(1-13)Online publication date: 20-May-2024
https://dl.acm.org/doi/10.1145/3597503.3639226
Korfmann RBeyersdorffer PMünch JKuhrmann M(2024)Using Data Augmentation to Support AI-Based Requirements Evaluation in Large-Scale ProjectsSystems, Software and Services Process Improvement10.1007/978-3-031-71139-8_7(97-111)Online publication date: 7-Sep-2024
https://doi.org/10.1007/978-3-031-71139-8_7
Hadi MFard F(2023)Evaluating pre-trained models for user feedback analysis in software engineering: a study on classification of app-reviewsEmpirical Software Engineering10.1007/s10664-023-10314-x28:4Online publication date: 23-May-2023
https://dl.acm.org/doi/10.1007/s10664-023-10314-x
Habib MWagner SGraziotin D(2021)Detecting Requirements Smells With Deep Learning: Experiences, Challenges and Future Work2021 IEEE 29th International Requirements Engineering Conference Workshops (REW)10.1109/REW53955.2021.00027(153-156)Online publication date: Sep-2021
https://doi.org/10.1109/REW53955.2021.00027
Subedi ISingh MRamasamy VWalia G(2021)Classification of Testable and Valuable User Stories by using Supervised Machine Learning Classifiers2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW53611.2021.00111(409-414)Online publication date: Oct-2021
https://doi.org/10.1109/ISSREW53611.2021.00111

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents