research-article

A Survey of Field-based Testing Techniques

Authors:

Antonia Bertolino,

Pietro Braione,

Guglielmo De Angelis,

Fitsum Kifetew,

Leonardo Mariani,

Roberto Pietrantuono,

Paolo TonellaAuthors Info & Claims

ACM Computing Surveys (CSUR), Volume 54, Issue 5

Article No.: 92, Pages 1 - 39

https://doi.org/10.1145/3447240

Published: 25 May 2021 Publication History

Abstract

Field testing refers to testing techniques that operate in the field to reveal those faults that escape in-house testing. Field testing techniques are becoming increasingly popular with the growing complexity of contemporary software systems. In this article, we present the first systematic survey of field testing approaches over a body of 80 collected studies, and propose their categorization based on the environment and the system on which field testing is performed. We discuss four research questions addressing how software is tested in the field, what is tested in the field, which are the requirements, and how field tests are managed, and identify many challenging research directions.

Supplementary Material

a92-bertolino-supp.pdf (bertolino.zip)

Supplemental movie, appendix, image and software files for, A Survey of Field-based Testing Techniques

Download
167.15 KB

References

[1]

Mohammed Akour, Akanksha Jaidev, and Tariq M. King. 2011. Towards change propagating test models in autonomic and adaptive systems. In Proceedings of the 18th International Conference on the Engineering of Computer-Based Systems (ECBS’11). IEEE, 89--96.

[2]

Midhat Ali, Francesco De Angelis, Daniele Fanì, Antonia Bertolino, Guglielmo De Angelis, and Andrea Polini. 2014. An extensible framework for online testing of choreographed services. Computer 47, 2 (Feb. 2014), 23--29.

Digital Library

[3]

Khaled Alnawasreh, Patrizio Pelliccione, Zhenxiao Hao, Mårten Rånge, and Antonia Bertolino. 2017. Online robustness testing of distributed embedded systems: An industrial approach. In Proceedings of the IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP’17). IEEE, 133--142.

Digital Library

[4]

Sultan Alyahya. 2020. Crowdsourced software testing: A systematic literature review. Inf. Softw. Technol. 127, Article 106363 (Nov. 2020).

[5]

Xiaoying Bai, Shufang Lee, Wei-Tek Tsai, and Yinong Chen. 2008. Ontology-based test modeling and partition testing of web services. In Proceedings of the IEEE International Conference on Web Services (ICWS’08). IEEE, 465--472.

Digital Library

[6]

Xiaoying Bai, Dezheng Xu, Guilan Dai, Wei-Tek Tsai, and Yinong Chen. 2007. Dynamic reconfigurable testing of service-oriented architecture. In Proceedings of the 31st Annual International Computer Software and Applications Conference (COMPSAC’07), Vol. 1. IEEE, 368--378.

Digital Library

[7]

Luciano Baresi and Carlo Ghezzi. 2010. The disappearing boundary between development-time and run-time. In Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research (FoSER’10). ACM, 17--22.

Digital Library

[8]

Cesare Bartolini, Antonia Bertolino, Sebastian Elbaum, and Eda Marchetti. 2011. Bringing white-box testing to Service Oriented Architectures through a Service Oriented Approach. J. Syst. Softw. 84, 4 (Apr. 2011), 655--668.

Digital Library

[9]

Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos engineering. IEEE Softw. 33, 3 (2016), 35--41.

Digital Library

[10]

Jonathan Bell, Christian Murphy, and Gail Kaiser. 2015. Metamorphic runtime checking of applications without test oracles. J. Defense Softw. Engineering 28, 2 (2015), 9--13.

[11]

Antonia Bertolino, Guglielmo De Angelis, Micael Gallego, Boni García, Francisco Gortázar, Francesca Lonetti, and Eda Marchetti. 2019. A systematic review on cloud testing. Comput. Surv. 52, 5, Article 93 (Sept. 2019).

[12]

Antonia Bertolino, Guglielmo De Angelis, Lars Frantzen, and Andrea Polini. 2007. The PLASTIC framework and tools for testing service-oriented applications. In International Summer Schools: Software Engineering (ISSSE 2006-2008), Andrea De Lucia and Filomena Ferrucci (Eds.). Lecture Notes in Computer Science, Vol. 5413. Springer, 106--139.

[13]

Antonia Bertolino, Guglielmo De Angelis, Sampo Kellomaki, and Andrea Polini. 2012. Enhancing service federation trustworthiness through online testing. Computer 45, 1 (2012), 66--72.

Digital Library

[14]

Antonia Bertolino, Guglielmo De Angelis, and Andrea Polini. 2012. Governance policies for verification and validation of service choreographies. In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST’12), J. Cordeiro and K.H. Krempels (Eds.), Lecture Notes in Business Information Processing, Vol. 140. Springer, 86--102.

[15]

Antonia Bertolino and Andrea Polini. 2009. SOA test governance: Enabling service integration testing across organization and technology borders. In Proceedings of the International Workshop on Web Testing (WebTest’09). IEEE, 277--286.

Digital Library

[16]

Smt. J. Sasi Bhanu, A. Vinaya Babu, and P. Trimurthy. 2015. A comprehensive architecture for dynamic evolution of online testing of an embedded system. J. Theor. Appl. Inf. Technol. 80, 1 (Oct. 2015), 160--172. http://www.jatit.org/volumes/Vol80No1/17Vol80No1.pdf

[17]

Jayaram Bobba, Weiwei Xiong, Luke Yen, Mark D. Hill, and David A. Wood. 2009. StealthTest: Low overhead online software testing using transactional memory. In Proceedings of the 18th International Conference on Parallel Architectures and Compilation Techniques (PACT’09). IEEE, 146--155.

[18]

Daniel Brenner, Colin Atkinson, Barbara Paech, Rainer Malaka, Matthias Merdes, and Dima Suliman. 2006. Reducing verification effort in component-based software engineering through built-in testing. In Proceedings of the 10th IEEE International Enterprise Distributed Object Computing Conference (EDOC’06). IEEE, 175--184.

Digital Library

[19]

Kai-Yuan Cai, Yong-Chao Li, and Ke Liu. 2004. Optimal and adaptive testing for software reliability assessment. Inf. Softw. Technol. 46, 15 (Dec. 2004), 989--1000.

[20]

Tien-Dung Cao, Patrick Félix, Richard Castanet, and Ismail Berrada. 2010. Online testing framework for web services. In Proceedings of the 3rd International Conference on Software Testing, Verification and Validation (ICST’10). IEEE, 363--372.

Digital Library

[21]

W. K. Chan, S. C. Cheung, and Karl R. P. H. Leung. 2007. A metamorphic testing approach for online testing of service-oriented software applications. Int. J. Web Serv. Res. 4, 2 (2007), 61--81.

[22]

Mark B. Cooray, James H. Hamlyn-Harris, and Robert G. Merkel. 2015. Dynamic test reconfiguration for composite web services. IEEE Trans. Serv. Comput. 8, 4 (2015), 576--585.

[23]

Andrew Diniz da Costa, Camila Nunes, Viviane Torres da Silva, Baldoino Fonseca, and Carlos J. P. de Lucena. 2010. JAAF+T: A framework to implement self-adaptive agents that apply self-test. In Proceedings of the 2010 ACM Symposium on Applied Computing (SAC’10). ACM, 928--935.

[24]

Huning Dai, Christian Murphy, and Gail Kaiser. 2010. Configuration fuzzing for software vulnerability detection. In Proceedings of the 5th International Conference on Availability, Reliability and Security (ARES’10). IEEE, 525--530.

[25]

Huning Dai, Christian Murphy, and Gail E. Kaiser. 2010. CONFU: Configuration fuzzing testing framework for software vulnerability detection. Int. J. Sec. Softw. Eng. 1, 3 (2010), 41--55.

Digital Library

[26]

Guglielmo De Angelis, Antonia Bertolino, and Andrea Polini. 2011. (role)CAST: A framework for on-line service testing. In Proceedings of the 7th International Conference on Web Information Systems and Technologies (WEBIST’11). SciTePress, Noordwijkerhout, The Netherlands, 13--18.

[27]

Guglielmo De Angelis, Antonia Bertolino, and Andrea Polini. 2012. Validation and verification policies for governance of service choreographies. In Proceedings of the 8th International Conference on Web Information Systems and Technologies (WEBIST’12). SciTePress, 58--70.

[28]

Vânia de Oliveira Neves, Márcio Eduardo Delamaro, and Paulo Cesar Masiero. 2014. An environment to support structural testing of autonomous vehicles. In Proceedings of the IV Brazilian Symposium on Computing Systems Engineering (SBESC’14). IEEE, 19--24.

Digital Library

[29]

Pouria Derakhshanfar, Xavier Devroey, Gilles Perrouin, Andy Zaidman, and Arie van Deursen. 2020. Search-based crash reproduction using behavioural model seeding. Softw. Test. Verif. Reliab. 30, 3, Article e1733 (May 2020).

[30]

Peter H. Deussen, George Din, and Ina Schieferdecker. 2002. An on-line test platform for component-based systems. In Proceedings of the 27th Annual NASA Goddard Software Engineering Workshop (SEW’02). IEEE, 96--103.

Digital Library

[31]

Peter H. Deussen, George Din, and Ina Schieferdecker. 2003. A TTCN-3 based online test and validation platform for Internet services. In Proceedings of the 6th International Symposium on Autonomous Decentralized Systems (ISADS’03). IEEE, 177--184.

Digital Library

[32]

Massimiliano Di Penta, Marcello Bruno, Gianpiero Esposito, Valentina Mazza, and Gerardo Canfora. 2007. Web Services Regression Testing. Springer, 205--234.

[33]

Dimitris Dranidis, Andreas Metzger, and Dimitrios Kourtesis. 2010. Enabling proactive adaptation through just-in-time testing of conversational services. In Proceedings of the 3rd European Conference (ServiceWave’10), Elisabetta Di Nitto and Ramin Yahyapour (Eds.), Lecture Notes in Computer Science,Vol. 6481. Springer, 63--75.

[34]

Sebastian Elbaum and Madeline Hardojo. 2004. An empirical study of profiling strategies for released software and their impact on testing activities. In Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’04). ACM, 65--75.

Digital Library

[35]

Brian Fitzgerald and Klaas-Jan Stol. 2017. Continuous software engineering: A roadmap and agenda. J. Syst. Softw. 123, 1 (Jan. 2017), 176--189.

[36]

Lars Frantzen, Jan Tretmans, and Tim A. C. Willemse. 2004. Test generation based on symbolic specifications. In Proceedings of 4th International Workshop on Formal Approaches to Software Testing (FATES’04), Jens Grabowski and Brian Nielsen (Eds.), Lecture Notes in Computer Science,Vol. 3395. Springer, 1--15.

[37]

Erik M. Fredericks and Betty H. C. Cheng. 2015. Automated generation of adaptive test plans for self-adaptive systems. In Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’15). IEEE, 157--167.

[38]

Erik M. Fredericks, Byron DeVries, and Betty H. C. Cheng. 2014. Towards run-time adaptation of test cases for self-adaptive systems in the face of uncertainty. In Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’14). ACM, 17--26.

[39]

Luca Gazzola, Leonardo Mariani, Fabrizio Pastore, and Mauro Pezzè. 2017. An exploratory study of field failures. In Proceedings of the 28th International Symposium on Software Reliability Engineering (ISSRE’17). IEEE, 67--77.

[40]

Okehee Goh and Yann-Hang Lee. 2007. Schedulable online testing framework for real-time embedded applications in VM. In Proceedings of the International Conference on Embedded and Ubiquitous Computing (EUC’07), Tei-Wei Kuo, Edwin Sha, Minyi Guo, Laurence T. Yang, and Zili Shao (Eds.), Lecture Notes in Computer Science,Vol. 4808. Springer, 730--741.

[41]

Alberto González, Eric Piel, and Hans-Gerhard Gross. 2008. Architecture support for runtime integration and verification of component-based systems of systems. In Proceedings of the International Workshop on Automated Engineering of Autonomous and Run-tiMe Evolving Systems (ARAMIS’08). IEEE, 41--48.

Digital Library

[42]

Alberto Gonzalez-Sanchez, Eric Piel, and Hans-Gerhard Gross. 2009. RiTMO: A method for runtime testability measurement and optimisation. In Proceedings of the 9th International Conference on Quality Software (QSIC’09). IEEE, 377--382.

Digital Library

[43]

Alberto Gonzalez-Sanchez, Eric Piel, Hans-Gerhard Gross, and Arjan J. C. van Gemund. 2010. Runtime testability in dynamic high-availability component-based systems. In Proceedings of the 2nd International Conference on Advances in System Testing and Validation Lifecycle (VALID’10). IEEE, 37--42.

[44]

Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Jian Lü, and Zhendong Su. 2016. Automatic runtime recovery via error handler synthesis. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE’16). ACM, 684--695.

Digital Library

[45]

Petr Hosek and Cristian Cadar. 2015. VARAN the unbelievable: An efficient n-version execution framework. In Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS’15), ACM SIGPLAN Notices, Vol. 50. ACM, 339--353.

Digital Library

[46]

Zhan-Wei Hui, Song Huang, and Meng-Yu Ji. 2016. A runtime-testing method for integer overflow detection based on metamorphic relations. J. Intell. Fuzzy Syst. 31, 4 (2016), 2349--2361.

Digital Library

[47]

Waldemar Hummer, Orna Raz, Onn Shehory, Philipp Leitner, and Schahram Dustdar. 2011. Test coverage of data-centric dynamic compositions in service-based systems. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation (ICST’11). IEEE, 40--49.

Digital Library

[48]

Waldemar Hummer, Orna Raz, Onn Shehory, Philipp Leitner, and Schahram Dustdar. 2013. Testing of data-centric and event-based dynamic service compositions. Softw. Test. Verif. Reliabil. 23, 6 (Sep. 2013), 465--497.

[49]

Melody Y. Ivory and Marti A. Hearst. 2001. The state of the art in automating usability evaluation of user interfaces. Comput. Surv. 33, 4 (Dec. 2001), 470--516.

Digital Library

[50]

Wei Jin and Alessandro Orso. 2012. BugRedux: Reproducing field failures for in-house debugging. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE, 474--484.

Digital Library

[51]

Katsumi Kawano, Masayuki Orimo, and Kinji Mori. 1989. Autonomous decentralized system test technique. In Proceedings of the 13th Annual International Computer Software & Applications Conference (COMPSAC’89). IEEE, 52--57.

[52]

Fitsum Meshesha Kifetew, Wei Jin, Roberto Tiella, Alessandro Orso, and Paolo Tonella. 2014. Reproducing field failures for programs with complex grammar-based input. In Proceedings of the 7th IEEE International Conference on Software Testing, Verification and Validation (ICST’14). IEEE, 163--172.

Digital Library

[53]

Tariq M. King, Andrew A. Allen, Rodolfo Cruz, and Peter J. Clarke. 2011. Safe runtime validation of behavioral adaptations in autonomic software. In Proceedings of the 8th International Conference on Autonomic and Trusted Computing (ATC’11), Jose M. Alcaraz Calero, Laurence T. Yang, Félix Gómez Mármol, and Luis Javier García Villalba (Eds.), Lecture Notes in Computer Science, Vol. 6906. Springer, 31--46.

[54]

Tariq M. King, Djuradj Babich, Jonatan Alava, Peter J. Clarke, and Ronald Stevens. 2007. Towards self-testing in autonomic computing systems. In Proceedings of the 8th International Symposium on Autonomous Decentralized Systems (ISADS’07). IEEE, 51--58.

Digital Library

[55]

Tariq M. King, Alain E. Ramirez, Rodolfo Cruz, and Peter J. Clarke. 2007. An integrated self-testing framework for autonomic computing systems. J. Comput. 2, 9 (2007), 37--49.

[56]

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE-2007-01.

[57]

Ron Kohavi and Roger Longbotham. 2017. Online controlled experiments and A/B testing. In Encyclopedia of Machine Learning and Data Mining, Claude Sammut and Geoffrey I. Webb (Eds.). Springer, 922--929.

[58]

Mariam Lahami, Fairouz Fakhfakh, Moez Krichen, and Mohamed Jmaiel. 2012. Towards a TTCN-3 test system for runtime testing of adaptable and distributed systems. In Proceedings of the 24th IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS’12), Brian Nielsen and Carsten Weise (Eds.), Lecture Notes in Computer Science,Vol. 7641. Springer, 71--86.

[59]

Mariam Lahami and Moez Krichen. 2013. Test isolation policy for safe runtime validation of evolvable software systems. In Proceedings of the Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE’13). IEEE, 377--382.

Digital Library

[60]

Mariam Lahami, Moez Krichen, Mariam Bouchakwa, and Mohamed Jmaiel. 2012. Using knapsack problem model to design a resource aware test architecture for adaptable and distributed systems. In Proceedings of the 24th IFIP WG 6.1 International Conference on Testing Software and Systems (ICTSS’12), Brian Nielsen and Carsten Weise (Eds.), Lecture Notes in Computer Science,Vol. 7641. Springer, 103--118.

[61]

Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2012. A distributed test architecture for adaptable and distributed real-time systems. In Avancées Récentes Dans le Domaine Des Architectures Logicielles: Articles Sélectionnés et Etendus de (CAL'11) (Revue des Nouvelles Technologies de l'Information), Philippe Aniorté (Ed.), Vol. L-6. Hermann, 732--92.

[62]

Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2013. Runtime testing framework for improving quality in dynamic service-based systems. In Proceedings of the 2nd International Workshop on Quality Assurance for Service-based Applications (QASBA’13). ACM, 17--24.

Digital Library

[63]

Mariam Lahami, Moez Krichen, and Mohamed Jmaïel. 2015. Runtime testing approach of structural adaptations for dynamic and distributed systems. Int. J. Comput. Appl. Technol. 51, 4 (Jul. 2015), 259--272.

Digital Library

[64]

Mariam Lahami, Moez Krichen, and Mohamed Jmaiel. 2016. Safe and efficient runtime testing framework applied in dynamic and distributed systems. Sci. Comput. Program. 122, 1 (2016), 1--28.

Digital Library

[65]

Gregory L. Lee, Dong H. Ahn, Dorian C. Arnold, Bronis R. de Supinski, Matthew Legendre, Barton P. Miller, Martin Schulz, and Ben Liblit. 2008. Lessons learned at 208K: Towards debugging millions of cores. In Proceedings of the 2008 ACM/IEEE Conference on Supercomputing (SC’08). IEEE, 1--9.

[66]

Kihwal Lee and Lui Sha. 2005. A dependable online testing and upgrade architecture for real-time embedded systems. In Proceedings of the 11th IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA’05). IEEE, 160--165.

Digital Library

[67]

Seoung-Hyeon Lee and Jung-Chan Na. 2017. Development of test agents for automated dynamic testing of UNIWAY. In Proceedings of the International Conference on Advanced Multimedia and Ubiquitous Engineering (MUE/FutureTech’17), James J. (Jong Hyuk) Park, Shu-Ching Chen, and Kim-Kwang Raymond Choo (Eds.), Lecture Notes in Electrical Engineering, Vol. 448. Springer, 683--688.

[68]

Martin Leucker and Christian Schallhart. 2009. A brief account of runtime verification. J. Logic Algebr. Program. 78, 5 (2009), 293--303.

[69]

Riku Luostarinen, Risto Järvinen, Juho Määttä, and Jukka Manner. 2016. A model for usability testing in challenging environments. In Proceedings of the International Conference on Military Communications and Information Systems (ICMCIS’16). IEEE, 1--6.

[70]

Bo Ma, Bin Chen, Xiaoying Bai, and Junfei Huang. 2010. Design of BDI agent for adaptive performance testing of web services. In Proceedings of the 10th International Conference on Quality Software (QSIC’10). IEEE, 435--440.

Digital Library

[71]

Afef Jmal Maâlej, Moez Krichen, and Mohamed Jmaïel. 2012. Model-based conformance testing of WS-BPEL compositions. In Proceedings of the 4th IEEE International Workshop on Software Testing Automation (STA’12). IEEE, 452--457.

Digital Library

[72]

Lijun Mei, W. K. Chan, T. H. Tse, Bo Jiang, and Ke Zhai. 2015. Preemptive regression testing of workflow-based web services. IEEE Trans. Serv. Comput. 8, 5 (Sep. 2015), 740--754.

[73]

Andreas Metzger, Osama Sammodi, Klaus Pohl, and Mark Rzepka. 2010. Towards pro-active adaptation with confidence: Augmenting service monitoring with online testing. In Proceedings of the 2010 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’10). ACM, 20--28.

Digital Library

[74]

Jesús Morán, Antonia Bertolino, Claudio de la Riva, and Javier Tuya. 2017. Towards ex Vivo testing of mapreduce applications. In Proceedings of the 2017 International Conference on Software Quality, Reliability and Security (QRS’17). IEEE, 73--80.

[75]

Christian Murphy, Gail E. Kaiser, Ian Vo, and Matt Chu. 2009. Quality assurance of software applications using the in Vivo testing approach. In Proceedings of the 2nd International Conference on Software Testing Verification and Validation (ICST’09). IEEE, 111--120.

Digital Library

[76]

Christian Murphy, Kuang Shen, and Gail Kaiser. 2009. Automatic system testing of programs without test oracles. In Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA’09). ACM, 189--200.

Digital Library

[77]

Christian Murphy, Moses Vaughan, Waseem Ilahi, and Gail Kaiser. 2010. Automatic detection of previously-unseen application states for deployment environment testing and analysis. In Proceedings of the 5th Workshop on Automation of Software Test (AST’10). ACM, 16--23.

Digital Library

[78]

Vânia de Oliveira Neves, Márcio Eduardo Delamaro, and Paulo Cesar Masiero. 2016. Combination and mutation strategies to support test data generation in the context of autonomous vehicles. Int. J. Embed. Syst. 8, 5/6 (2016), 464--482.

[79]

Vânia de Oliveira Neves, Márcio Eduardo Delamaro, and Paulo Cesar Masiero. 2017. Automated structural software testing of autonomous vehicles. In Proceedings of the 20th Iberoamerican Conference on Software Engineering (CibSE’17), Vol. 1. Curran Associates, Buenos Aires, Argentina, 15--28.

[80]

Dirk Niebuhr, Andreas Rausch, Cornel Klein, Juergen Reichmann, and Reiner N. Schmid. 2009. Achieving dependable component bindings in dynamic adaptive systems - A runtime testing approach. In Proceedings of the 3rd IEEE International Conference on Self-Adaptive and Self-Organizing Systems (SASO’09). IEEE, 186--197.

[81]

Eiji Nishijima, Hiroshi Yamamoto, Katsumi Kawano, Kazunori Fujiwara, and Keiji Oshima. 1996. On-line testing for application software of widely distributed system. In Proceedings of the 15th Symposium on Reliable Distributed Systems (SRDS’96). IEEE, 54--63.

[82]

Marc Oriol, Xavier Franch, and Jordi Marco. 2015. Monitoring the service-based system lifecycle with SALMon. Expert Syst. Appl. 42, 19 (Nov. 2015), 6507--6521.

Digital Library

[83]

Youngki Park, Woosung Jung, Byungjeong Lee, and Chisu Wu. 2009. Automatic discovery of web services based on dynamic black-box testing. In Proceedings of the 33rd Annual IEEE International Computer Software and Applications Conference (COMPSAC’09), Vol. 1. IEEE, 107--114.

Digital Library

[84]

Kai Petersen, Sairam Vakkalanka, and Ludwik Kuzniarz. 2015. Guidelines for conducting systematic mapping studies in software engineering: An update. Inf. Softw. Technol. 64 (Aug. 2015), 1--18.

Digital Library

[85]

Mauro Pezzè and Michal Young. 2008. Software Testing and Analysis: Process, Principles, and Techniques. John Wiley & Sons.

[86]

Éric Piel and Alberto Gonzalez-Sanchez. 2009. Data-flow integration testing adapted to runtime evolution in component-based systems. In Proceedings of the 2009 ESEC/FSE workshop on Software Integration and Evolution @ Runtime (SINTER’09). ACM, Amsterdam, The Netherlands, 3--10.

Digital Library

[87]

Adam Porter, Cemal Yilmaz, Atif M. Memon, Douglas C. Schmidt, and Bala Natarajan. 2007. Skoll: A process and infrastructure for distributed continuous quality assurance. IEEE Trans. Softw. Eng. 33, 8 (2007), 510--525.

Digital Library

[88]

Jeremias Rößler, Andreas Zeller, Gordon Fraser, Cristian Zamfir, and George Candea. 2013. Reconstructing core dumps. In Proceedings of the IEEE 6th International Conference on Software Testing, Verification and Validation, (ICST’13). IEEE, 114--123.

Digital Library

[89]

Osama Sammodi, Andreas Metzger, Xavier Franch, Marc Oriol, Jordi Marco, and Klaus Pohl. 2011. Usage-based online testing for proactive adaptation of service-based applications. In Proceedings of the 35th Annual IEEE International Computer Software and Applications Conference (COMPSAC’11). IEEE, 582--587.

Digital Library

[90]

Gerald Schermann, Jürgen Cito, and Philipp Leitner. 2018. Continuous experimentation: Challenges, implementation techniques, and current research. IEEE Softw. 35, 2 (2018), 26--31.

[91]

Carol Smidts, Chetan Mutha, Manuel Rodriguez, and Matthew J. Gerber. 2014. Software testing with an operational profile: OP definition. Comput. Surv. 46, 3, Article 39 (Feb. 2014).

[92]

Mozhan Soltani, Pouria Derakhshanfar, Xavier Devroey, and Arie van Deursen. 2020. A benchmark-based evaluation of search-based crash reproduction. Empir. Softw. Eng. 25, 1 (Jan. 2020), 96--138.

Digital Library

[93]

Ronald Stevens, Brittany Parsons, and Tariq M. King. 2007. A self-testing autonomic container. In Proceedings of the 45th Annual Southeast Regional Conference (ACM-SE 45). ACM, 1--6.

[94]

Dima Suliman, Barbara Paech, Lars Borner, Colin Atkinson, Daniel Brenner, Matthias Merdes, and Rainer Malaka. 2006. The MORABIT approach to runtime component testing. In Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Vol. 2. IEEE, 171--176.

Digital Library

[95]

Jüri Vain, Leonidas Tsiopoulos, Vyacheslav Kharchenko, Apneet Kaur, Maksim Jenihhin, and Jaan Raik. 2017. Multi-fragment Markov model guided online test generation for MPSoC. In Proceedings of the 3rd International Workshop on Theory of Reliability and Markov Modeling for Information Technologies (TheRMIT’17). 594--607.

[96]

Margus Veanes, Colin Campbell, Wolfram Schulte, and Nikolai Tillmann. 2005. Online testing with model programs. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE-13), ACM SIGSOFT Software Engineering Notes,Vol. 30. ACM, 273--282.

Digital Library

[97]

Arnold P. O. S. Vermeeren, Effie Lai-Chong Law, Virpi Roto, Marianna Obrist, Jettie Hoonhout, and Kaisa Väänänen-Vainio-Mattila. 2010. User experience evaluation methods: Current state and development needs. In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries (NordiCHI’10). ACM, 521--530.

Digital Library

[98]

Yongbo Wang, Xiaoying Bai, Juanzi Li, and Ruobo Huang. 2007. Ontology-based test case generation for testing web services. In Proceedings of the 8th International Symposium on Autonomous Decentralized Systems (ISADS’07). IEEE, 43--50.

Digital Library

[99]

Yingxu Wang, Graham King, Dilip Patel, Shushma Patel, and Alec Dorling. 1999. On coping with real-time software dynamic inconsistency by built-in tests. Ann. Softw. Eng. 7, 1 (Oct. 1999), 283--296.

Digital Library

[100]

Hiroshi Yamamoto, Akira Yoshizawa, Katsumi Kawano, Kinji Mori, Keiji Oshima, and Tsunao Kawamura. 1993. On-line software test techniques based on autonomous decentralized system. In Proceedings of the 4th Workshop on Future Trends of Distributed Computing Systems (FTDCS’93). IEEE, 291--296.

[101]

Chunyang Ye and Hans-Arno Jacobsen. 2013. Whitening SOA testing via event exposure. IEEE Trans. Softw. Eng. 39, 10 (2013), 1444--1465.

Digital Library

[102]

Jia Zhang. 2004. An approach to facilitate reliability testing of web services components. In Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). IEEE, 210--218.

Digital Library

[103]

Hong Zhu. 2006. A framework for service-oriented testing of web services. In Proceedings of the 3rd International Workshop on Quality Assurance and Testing Web-Based Applications (QATWBA’06), Vol. 2. IEEE, 145--150.

Digital Library

[104]

Hong Zhu and Yufeng Zhang. 2012. Collaborative testing of web services. IEEE Trans. Serv. Comput. 5, 1 (2012), 116--130.

Digital Library

[105]

Hong Zhu and Yufeng Zhang. 2014. A test automation framework for collaborative testing of web service dynamic compositions. In Advanced Web Services, Athman Bouguettaya, Quan Z. Sheng, and Florian Daniel (Eds.). Springer, 171--197.

Cited By

Li JZhang MLi NWeyns DJin ZTei K(2024)Generative AI for Self-Adaptive Systems: State of the Art and Research RoadmapACM Transactions on Autonomous and Adaptive Systems10.1145/368680319:3(1-60)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3686803
Qu DZhao CJiang YXu C(2024)Towards Life-long Software Self-validation in ProductionProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671382(357-366)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671382
Silva SPelliccione PBertolino A(2024)Self-Adaptive Testing in the FieldACM Transactions on Autonomous and Adaptive Systems10.1145/362716319:1(1-37)Online publication date: 14-Feb-2024
https://dl.acm.org/doi/10.1145/3627163
Show More Cited By

Index Terms

A Survey of Field-based Testing Techniques
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Flakiness goes live: Insights from an In Vivo testing simulation study
Abstract Context:
Test flakiness is a topmost concern in software test automation. While conducting pre-deployment testing, those tests that are flagged as flaky are put aside for being either repaired or discarded.
Objective:
We hypothesise that some ...
Fault-based test suite prioritization for specification-based testing

Context: Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in ...
The experimental applications of search-based techniques for model-based testing

Graphical abstractDisplay Omitted HighlightsA systematic review of applications of search-based techniques for model-based testing is provided.Four taxonomies are proposed to classify the applications based on the purpose, problems, solutions and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 54, Issue 5

June 2022

719 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3467690

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 May 2021

Accepted: 01 January 2021

Revised: 01 January 2021

Received: 01 April 2020

Published in CSUR Volume 54, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Italian MIUR PRIN 2015
European Research Council

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
685
Total Downloads

Downloads (Last 12 months)116
Downloads (Last 6 weeks)19

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li JZhang MLi NWeyns DJin ZTei K(2024)Generative AI for Self-Adaptive Systems: State of the Art and Research RoadmapACM Transactions on Autonomous and Adaptive Systems10.1145/368680319:3(1-60)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3686803
Qu DZhao CJiang YXu C(2024)Towards Life-long Software Self-validation in ProductionProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3671382(357-366)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3671382
Silva SPelliccione PBertolino A(2024)Self-Adaptive Testing in the FieldACM Transactions on Autonomous and Adaptive Systems10.1145/362716319:1(1-37)Online publication date: 14-Feb-2024
https://dl.acm.org/doi/10.1145/3627163
Caldas RGarcía JSchiopu MPelliccione PRodrigues GBerger T(2024)Runtime Verification and Field-Based Testing for ROS-Based Robotic SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.344469750:10(2544-2567)Online publication date: 19-Aug-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3444697
Cornejo OBriola DMicucci DGinelli DMariani LSantos Parrilla AJuristo N(2024)A family of experiments about how developers perceive delayed system response timeSoftware Quality Journal10.1007/s11219-024-09660-w32:2(567-605)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1007/s11219-024-09660-w
Mandrioli CNyberg Carlsson MMaggio M(2023)Testing Abstractions for Cyber-Physical Control SystemsACM Transactions on Software Engineering and Methodology10.1145/361717033:1(1-32)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3617170
Silva S(2023)Reviewing and Proposing Approaches for Self-adaptive Testing in the Field2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW60843.2023.00035(13-16)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSREW60843.2023.00035
Dakkak ABosch JOlsson HIssa Mattos D(2023)Continuous deployment in software-intensive system-of-systemsInformation and Software Technology10.1016/j.infsof.2023.107200159:COnline publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1016/j.infsof.2023.107200
Malik JPastore F(2023)An empirical study of vulnerabilities in edge frameworks to support security testing improvementEmpirical Software Engineering10.1007/s10664-023-10330-x28:4Online publication date: 5-Jul-2023
https://dl.acm.org/doi/10.1007/s10664-023-10330-x
Pelliccione PBuhnova BGottschalk SWeber IEngels G(2023)Architecting and Engineering Value-Based EcosystemsSoftware Architecture10.1007/978-3-031-36847-9_3(41-68)Online publication date: 3-Jun-2023
https://doi.org/10.1007/978-3-031-36847-9_3
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents