Article

A case study on testing, commissioning, and operation of very-large-scale software systems

Authors:

Michael Vierhauser,

Paul GrünbacherAuthors Info & Claims

ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

Pages 125 - 134

https://doi.org/10.1145/2591062.2591179

Published: 31 May 2014 Publication History

Abstract

An increasing number of software systems today are very-large-scale software systems (VLSS) with system-of-systems (SoS) architectures. Due to their heterogeneity and complexity VLSS are difficult to understand and analyze, which results in various challenges for development and evolution. Existing software engineering processes, methods, and tools do not sufficiently address the characteristics of VLSS. Also, there are only a few empirical studies on software engineering for VLSS. We report on results of an exploratory case study involving engineers and technical project managers of an industrial automation VLSS for metallurgical plants. The paper provides empirical evidence on how VLSS are tested, commissioned, and operated in practice. The paper discusses practical challenges and reports industrial requirements regarding process and tool support. In particular, software processes and tools need to provide general guidance at the VLSS level as well as specific methods and tools for systems that are part of the VLSS. Processes and tools need to support multi-disciplinary engineering across system boundaries. Furthermore, managing variability and evolution is success-critical in VLSS verification and validation.

References

[1]

S. Adolph, W. Hall, and P. Kruchten. Using grounded theory to study the experience of software development. Empirical Software Engineering, 16(4):487–513, 2011.

Digital Library

[2]

B. Boehm. Some future trends and implications for systems and software engineering processes. Systems Engineering, 9(1):1–19, 2006.

[3]

B. Boehm and J. A. Lane. 21st century processes for acquiring 21st century software-intensive systems of systems. Crosstalk: Journal of Defence Software Engineering, 19(5):4–9, 2006.

[4]

B. W. Boehm. The changing nature of software evolution. IEEE Software, 27(4):26–29, 2010.

Digital Library

[5]

B. W. Boehm and J. A. Lane. New processes for new horizons: the incremental commitment model. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, pages 501–502. ACM, 2010.

Digital Library

[6]

J. Bézivin, F. Jouault, and P. Valduriez. On the need for megamodels. In Workshop on Best Practices for Model-Driven Software Development: Proceedings of the 19th Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 1–9. ACM, 2004.

[7]

D. S. Caffall and J. B. Michael. Architectural framework for a system-of-systems. In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 1876–1881. IEEE, 2005.

[8]

M. B. Chrissis, M. Konrad, and S. Shrum. CMMI for development: Guidelines for process integration and product improvement. Pearson Education, 2011.

Digital Library

[9]

J. Dahmann, J. A. Lane, G. Rebovich, and R. Lowry. Systems of systems test and evaluation challenges. In 5th International Conference on System of Systems Engineering (SoSE 2010), pages 1–6, 2010.

[10]

T. Dyb˚ a and T. Dingsøyr. Empirical studies of agile software development: A systematic review. Information & Software Technology, 50(9-10):833––859, 2008.

Digital Library

[11]

W. Emam, K. E. Melo and J.-N. Drouin. SPICE: The theory and practice of software process improvement and capability determination. IEEE Computer Society Press, 1997.

Digital Library

[12]

E. Engström and P. Runeson. Software product line testing: systematic mapping study. Information & Software Technology, 53(1):2––13, 2011.

Digital Library

[13]

B. G. Glaser and A. L. Strauss. The discovery of grounded theory: Strategies for qualitative research. Transaction Books, 2009.

[14]

G. Holl, P. Grünbacher, and R. Rabiser. A systematic review and an expert survey on capabilities supporting multi product lines. Information & Software Technology, 54(8):828––852, 2012.

Digital Library

[15]

M. N. Huhns and M. P. Singh. Service-oriented computing: Key concepts and principles. IEEE Internet Computing, 9(1):75–81, 2005.

Digital Library

[16]

T. Kähkönen. Agile methods for large organizations – building communities of practice. In Agile Development Conference (ADC’04), pages 2–11. IEEE Computer Society, 2004.

Digital Library

[17]

M. M. Lehman. Laws of software evolution revisited. In Proceedings 5th European Software Process Technology Workshop (EWSPT’96), volume 1149 of Lecture Notes in Computer Science, pages 108–124. Springer, 1996.

Digital Library

[18]

M. Lindvall, D. Muthig, A. Dagnino, C. Wallin, M. Stupperich, D. Kiefer, J. May, and T. Kähkönen. Agile software development in large organizations. IEEE Computer, 37(12):26–34, 2004.

Digital Library

[19]

M. W. Maier. Architecting principles for systems-of-systems. Systems Engineering, 1(4):267–284, 1998.

[20]

K. Manikas and K. M. Hansen. Software ecosystems: a systematic literature review. Journal of Systems and Software, 86(5):1294–1306, 2013.

Digital Library

[21]

T. Moser and S. Biﬄ. Semantic integration of software and systems engineering environments. IEEE Transactions on Systems, Man, and Cybernetics, 42(1):38–50, 2012.

Digital Library

[22]

T. Moser, S. Biﬄ, W. D. Sunindyo, and D. Winkler. Integrating production automation expert knowledge across engineering domains. International Journal of Distributed Systems and Technologies (IJDST), 2(3):88–103, 2011.

Digital Library

[23]

L. Northrop, P. Feiler, R. P. Gabriel, J. Goodenough, R. Linger, T. Longstaff, R. Kazman, M. Klein, D. Schmidt, K. Sullivan, and K. Wallnau. Ultra-Large-Scale Systems – The Software Challenge of the Future. Carnegie Mellon University, 2006.

[24]

A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large software systems. In Proceedings of the 12th International Symposium on the Foundations of Software Engineering, pages 241–251. ACM, 2004.

Digital Library

[25]

D. Pagano and B. Brügge. User involvement in software evolution practice: a case study. In 35th International Conference on Software Engineering, pages 953–962. IEEE / ACM, 2013.

Digital Library

[26]

K. Pohl, G. Böckle, and F. van der Linden. Software product line engineering: foundations, principles, and techniques. Springer, 2005.

Digital Library

[27]

K. Pohl and A. Metzger. Software product line testing. Commun. ACM, 49(12):78–81, 2006.

Digital Library

[28]

C. Robson. Real World Research. Wiley & Sons, 2011.

[29]

P. Runeson and M. Höst. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14(2):131–164, 2009.

Digital Library

[30]

H. P. Siy and D. E. Perry. Challenges in evolving a large scale software product. In Principles of Software Evolution Workshop. 1998 International Software Engineering Conference, 1998.

[31]

Software & Systems Process Engineering Metamodel Specification (SPEM) Version 2.0, April 2008. http://www.omg.org/spec/SPEM/2.0/, 2008.

[32]

K. Sullivan. Ultra-large-scale systems (keynote talk). In 12th International Symposium on Component Based Software Engineering (CBSE’09), 2009.

[33]

M. Vierhauser, R. Rabiser, P. Grünbacher, C. Danner, and S. Wallner. Evolving systems of systems: industrial challenges and research perspectives. In Proceedings of the First International Workshop on Software Engineering for Systems-of-Systems, pages 1–4. ACM, 2013.

Digital Library

[34]

C. Wallin, F. Ekdahl, and S. Larsson. Integrating business and software development models. IEEE Software, 19(6):28–33, 2002.

Digital Library

[35]

R. Wettel and M. Lanza. Visual exploration of large-scale system evolution. In Proceedings of the 15th Working Conference on Reverse Engineering, pages 219–228. IEEE, 2008.

Digital Library

[36]

C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, and B. Regnell. Experimentation in Software Engineering. Springer, 2012.

Digital Library

[37]

R. K. Yin. Case study research: Design and methods, volume 5. Sage, 2009.

Cited By

Enberg TAlander SPaasivaara M(2024)Onboarding for an Agile Software Development CompanyAgile Processes in Software Engineering and Extreme Programming10.1007/978-3-031-61154-4_10(144-162)Online publication date: 31-May-2024
https://doi.org/10.1007/978-3-031-61154-4_10
Elvira TCouder JProcko TOchoa O(2023)Impacts of Catastrophic Forgetting: From a Machine Learning Engineering Perspective2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE)10.1109/CSCE60160.2023.00010(01-08)Online publication date: 24-Jul-2023
https://doi.org/10.1109/CSCE60160.2023.00010
Dobslaw FWan RHao Y(2023)Generic and industrial scale many-criteria regression test selectionJournal of Systems and Software10.1016/j.jss.2023.111802205(111802)Online publication date: Nov-2023
https://doi.org/10.1016/j.jss.2023.111802
Show More Cited By

Index Terms

A case study on testing, commissioning, and operation of very-large-scale software systems
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software notations and tools
    1. Software configuration management and version control systems

Recommendations

A Requirements Monitoring Infrastructure for Very-Large-Scale Software Systems
REFSQ 2014: Proceedings of the 20th International Working Conference on Requirements Engineering: Foundation for Software Quality - Volume 8396

[Context and motivation] Approaches for requirements monitoring check the compliance of systems with their requirements during operation. [Question/problem] Despite many advances, requirements monitoring remains challenging particularly for very-large-...
Needs and challenges for a platform to support large-scale requirements engineering: a multiple-case study
ESEM '18: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Background: Requirement engineering is often considered a critical activity in system development projects. The increasing complexity of software as well as number and heterogeneity of stakeholders motivate the development of methods and tools for ...
Are you biting off more than you can chew? A case study on causes and effects of overscoping in large-scale software engineering

Context: Scope management is a core part of software release management and often a key factor in releasing successful software products to the market. In a market-driven case, when only a few requirements are known a priori, the risk of overscoping may ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

May 2014

741 pages

ISBN:9781450327688

DOI:10.1145/2591062

General Chair:
Pankaj Jalote
IIIT-Delhi, India
,
Program Chairs:
Lionel Briand
University of Luxembourg, Luxembourg
,
André van der Hoek
University of California, Irvine, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 7, 2014

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
426
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)2

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Enberg TAlander SPaasivaara M(2024)Onboarding for an Agile Software Development CompanyAgile Processes in Software Engineering and Extreme Programming10.1007/978-3-031-61154-4_10(144-162)Online publication date: 31-May-2024
https://doi.org/10.1007/978-3-031-61154-4_10
Elvira TCouder JProcko TOchoa O(2023)Impacts of Catastrophic Forgetting: From a Machine Learning Engineering Perspective2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE)10.1109/CSCE60160.2023.00010(01-08)Online publication date: 24-Jul-2023
https://doi.org/10.1109/CSCE60160.2023.00010
Dobslaw FWan RHao Y(2023)Generic and industrial scale many-criteria regression test selectionJournal of Systems and Software10.1016/j.jss.2023.111802205(111802)Online publication date: Nov-2023
https://doi.org/10.1016/j.jss.2023.111802
Fischer SKlammer CFernández ARabiser RRamler R(2023)Testing of highly configurable cyber–physical systems — Results from a two-phase multiple case studyJournal of Systems and Software10.1016/j.jss.2023.111624199(111624)Online publication date: May-2023
https://doi.org/10.1016/j.jss.2023.111624
Vierhauser MEgyed A(2023)Runtime Monitoring for Systems of SystemDigital Transformation10.1007/978-3-662-65004-2_8(203-222)Online publication date: 3-Feb-2023
https://doi.org/10.1007/978-3-662-65004-2_8
Vogel-Heuser BNeumann EFischer JMarcos MEstevez EBarbieri GSonnleithner LRabiser R(2022)Automation Software Architecture in CPPS - Definition, Challenges and Research Potentials2022 IEEE 5th International Conference on Industrial Cyber-Physical Systems (ICPS)10.1109/ICPS51978.2022.9816893(01-08)Online publication date: 24-May-2022
https://doi.org/10.1109/ICPS51978.2022.9816893
Fischer SRamler RKlammer CRabiser R(2021)Testing of Highly Configurable Cyber-Physical Systems – A Multiple Case StudyProceedings of the 15th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3442391.3442411(1-10)Online publication date: 9-Feb-2021
https://dl.acm.org/doi/10.1145/3442391.3442411
Verma AChoudhary ATiwari S(2021)Regression Test Case Selection: A Comparative Analysis of Metaheuristic AlgorithmsProceedings of Second Doctoral Symposium on Computational Intelligence10.1007/978-981-16-3346-1_49(605-615)Online publication date: 20-Sep-2021
https://doi.org/10.1007/978-981-16-3346-1_49
Muscarella SOsaisai MSheard S(2021)Systems and Software Interface SurveyINSIGHT10.1002/inst.1233424:2(9-16)Online publication date: 6-Aug-2021
https://doi.org/10.1002/inst.12334
Wang JDing YXiong NYeh WWang J(2020)GSCS: General Secure Consensus Scheme for Decentralized Blockchain SystemsIEEE Access10.1109/ACCESS.2020.30079388(125826-125848)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3007938
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents