research-article

Interval change-point detection for runtime probabilistic model checking

Authors:

Radu Calinescu,

Simos Gerasimou,

David FlynnAuthors Info & Claims

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Pages 163 - 174

https://doi.org/10.1145/3324884.3416565

Published: 27 January 2021 Publication History

Abstract

Recent probabilistic model checking techniques can verify reliability and performance properties of software systems affected by parametric uncertainty. This involves modelling the system behaviour using interval Markov chains, i.e., Markov models with transition probabilities or rates specified as intervals. These intervals can be updated continually using Bayesian estimators with imprecise priors, enabling the verification of the system properties of interest at runtime. However, Bayesian estimators are slow to react to sudden changes in the actual value of the estimated parameters, yielding inaccurate intervals and leading to poor verification results after such changes. To address this limitation, we introduce an efficient interval change-point detection method, and we integrate it with a state-of-the-art Bayesian estimator with imprecise priors. Our experimental results show that the resulting end-to-end Bayesian approach to change-point detection and estimation of interval Markov chain parameters handles effectively a wide range of sudden changes in parameter values, and supports runtime probabilistic model checking under parametric uncertainty.

References

[1]

Samaneh Aminikhanghahi and Diane J. Cook. 2017. A survey of methods for time series change point detection. Knowledge and Information Systems 51, 2 (2017), 339--367.

Digital Library

[2]

Adnan Aziz, Kumud Sanwal, Vigyan Singhal, and Robert Brayton. 1996. Verifying continuous time Markov chains. In Computer Aided Verification (LNCS), Rajeev Alur and Thomas A. Henzinger (Eds.), Vol. 1102. Springer Berlin Heidelberg, 269--276.

[3]

Christel Baier and Joost-Pieter Katoen. 2008. Principles of model checking. MIT press.

Digital Library

[4]

Christel Baier, Joost-Pieter Katoen, and Holger Hermanns. 1999. Approximative Symbolic Model Checking of Continuous-Time Markov Chains. In CONCUR'99 Concurrency Theory (LNCS), Jos C. M. Baeten and Sjouke Mauw (Eds.), Vol. 1664. Springer, Berlin, Heidelberg, 146--161.

[5]

Claudie Beaulieu, Jie Chen, and Jorge L Sarmiento. 2012. Change-point analysis as a tool to detect abrupt climate variations. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 370, 1962 (2012), 1228--1249.

[6]

James O Berger, Elías Moreno, Luis Raul Pericchi, M Jesús Bayarri, José M Bernardo, Juan A Cano, Julián De la Horra, Jacinto Martín, David Ríos-Insúa, Bruno Betrò, et al. 1994. An overview of robust Bayesian analysis. Test 3, 1 (1994), 5--124.

[7]

Jose M. Bernardo and Adrian F. M. Smith. 1994. Bayesian theory. Wiley.

[8]

Andrea Bianco and Luca de Alfaro. 1995. Model checking of probabilistic and nondeterministic systems. In Foundations of Software Technology and Theoretical Computer Science (LNCS), P. S. Thiagarajan (Ed.), Vol. 1026. Springer Berlin Heidelberg, 499--513.

[9]

Radu Calinescu, Marco Autili, Javier Cámara, Antinisca Di Marco, Simos Gerasimou, Paola Inverardi, Alexander Perucci, Nils Jansen, Joost-Pieter Katoen, Marta Kwiatkowska, et al. 2017. Synthesis and verification of self-aware computing systems. In Self-Aware Computing Systems. Springer, 337--373.

[10]

Radu Calinescu, Milan Češka, Simos Gerasimou, Marta Kwiatkowska, and Nicola Paoletti. 2018. Efficient synthesis of robust models for stochastic systems. Journal of Systems and Software 143 (2018), 140--158.

[11]

Radu Calinescu, Simos Gerasimou, Kenneth Johnson, and Colin Paterson. 2017. Using runtime quantitative verification to provide assurance evidence for self-adaptive software. In Software Engineering for Self-Adaptive Systems III. Assurances. Springer, 223--248.

[12]

Radu Calinescu, Carlo Ghezzi, Kenneth Johnson, Pezzé Mauro, Yasmin Rafiq, and Giordano Tamburrelli. 2016. Formal verification with confidence intervals to establish quality of service properties of software systems. IEEE Transactions on Reliability 65, 1 (2016), 107--125.

[13]

Radu Calinescu, Carlo Ghezzi, Marta Kwiatkowska, and Raffaela Mirandola. 2012. Self-adaptive software needs quantitative verification at runtime. Communication of the ACM 55, 9 (2012), 69--77.

Digital Library

[14]

Radu Calinescu, Raffaela Mirandola, Diego Perez-Palacin, and Danny Weyns. 2020. Understanding Uncertainty in Self-adaptive Systems. In 1st IEEE International Conference on Autonomic Computing and Self-Organizing Systems.

[15]

Radu Calinescu, Yasmin Rafiq, Kenneth Johnson, and Mehmet Emin Bakir. 2014. Adaptive model learning for continual verification of non-functional properties. In Proc. of the 5th ACM/SPEC Int. Conference on Performance Engineering (ICPE '14). ACM, New York, NY, USA, 87--98.

Digital Library

[16]

Radu Calinescu, Danny Weyns, Simos Gerasimou, Muhammad Usman Iftikhar, Ibrahim Habli, and Tim Kelly. 2018. Engineering trustworthy self-adaptive software with dynamic assurance cases. IEEE Transactions on Software Engineering 44, 11 (Nov. 2018), 1039--1069.

[17]

Bradley P. Carlin, Alan E. Gelfand, and Adrian F. M. Smith. 1992. Hierarchical Bayesian analysis of change-point problems. Journal of the Royal Statistical Society: Series C (Applied Statistics) 41, 2 (1992), 389--405.

[18]

Milan Ceška, Petr Pilař, Nicola Paoletti, Luboš Brim, and Marta Kwiatkowska. 2016. PRISM-PSY: Precise GPU-accelerated parameter synthesis for stochastic systems. In Tools and Algorithms for the Construction and Analysis of Systems (LNCS), Marsha Chechik and Jean-François Raskin (Eds.), Vol. 9636. Springer, Berlin, Heidelberg, 367--384.

Digital Library

[19]

Souymodip Chakraborty and Joost-Pieter Katoen. 2015. Model checking of open interval Markov chains. In Analytical and Stochastic Modelling Techniques and Applications (LNCS), Marco Gribaudo, Daniele Manini, and Anne Remke (Eds.), Vol. 9081. Springer, Cham, 30--42.

[20]

David Daly, William Brown, Henrik Ingo, Jim O'Leary, and David Bradford. 2020. The use of change point detection to identify software performance regressions in a continuous integration system. In Proc. of the ACM/SPEC International Conference on Performance Engineering (ICPE '20). ACM, New York, NY, USA, 67--75.

Digital Library

[21]

Christian Dehnert, Sebastian Junges, Joost-Pieter Katoen, and Matthias Volk. 2017. A Storm is coming: A modern probabilistic model checker. In Computer Aided Verification (LNCS), Rupak Majumdar and Viktor Kunčak (Eds.), Vol. 10427. Springer, Cham, 592--600.

[22]

Therese M. Donovan and Ruth M. Mickey. 2019. MCMC diagnostic approaches. In Bayesian Statistics for Beginners. Oxford University Press, Oxford.

[23]

Therese M. Donovan and Ruth M. Mickey. 2019. The white house problem revisited: MCMC with the Metropolis-Hastings algorithm. In Bayesian Statistics for Beginners. Oxford University Press, Oxford.

[24]

Ilenia Epifani, Carlo Ghezzi, Raffaela Mirandola, and Giordano Tamburrelli. 2009. Model evolution by run-time parameter adaptation. In Proc. of the 31st International Conference on Software Engineering (ICSE '09). IEEE, Washington, DC, USA, 111--121.

Digital Library

[25]

Ilenia Epifani, Carlo Ghezzi, and Giordano Tamburrelli. 2010. Change-point detection for black-box services. In Proc. of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE '10). ACM, New York, NY, USA, 227--236.

Digital Library

[26]

Michael Evans and Hadas Moshonov. 2006. Checking for prior-data conflict. Bayesian Analysis 1, 4 (2006), 893--914.

[27]

Antonio Filieri, Carlo Ghezzi, and Giordano Tamburrelli. 2012. A formal approach to adaptive software: Continuous assurance of non-functional requirements. Formal Aspects of Computing 24, 2 (2012), 163--186.

Digital Library

[28]

Antonio Filieri, Lars Grunske, and Alberto Leva. 2015. Lightweight adaptive filtering for efficient learning and updating of probabilistic models. In Proc. of the 37th International Conference on Software Engineering (ICSE '15). IEEE, Florence, Italy, 200--211.

[29]

A. Filieri, G. Tamburrelli, and C. Ghezzi. 2016. Supporting self-adaptation via quantitative verification and sensitivity analysis at run time. IEEE Transactions on Software Engineering 42, 1 (2016), 75--99.

Digital Library

[30]

Simos Gerasimou, Radu Calinescu, and Alec Banks. 2014. Efficient runtime quantitative verification using caching, lookahead, and nearly-optimal reconfiguration. In Proceedings of the 9th international symposium on software engineering for adaptive and self-managing systems. 115--124.

Digital Library

[31]

Simos Gerasimou, Radu Calinescu, Stepan Shevtsov, and Danny Weyns. 2017. UNDERSEA: an exemplar for engineering self-adaptive unmanned underwater vehicles. In IEEE/ACM 12th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. Buenos Aires, Argentina, 83--89.

Digital Library

[32]

Simos Gerasimou, Radu Calinescu, and Giordano Tamburrelli. 2018. Synthesis of probabilistic models for quality-of-service software engineering. Automated Software Engineering 25, 4 (2018), 785--831.

[33]

Hans Hansson and Bengt Jonsson. 1994. A logic for reasoning about time and reliability. Formal Aspects of Computing 6, 5 (Sept. 1994), 512--535.

Digital Library

[34]

Jeffrey O Kephart and David M Chess. 2003. The vision of autonomic computing. Computer 36, 1 (2003), 41--50.

Digital Library

[35]

Igor O. Kozine and Lev V. Utkin. 2002. Interval-valued finite Markov chains. Reliable Computing 8, 2 (2002), 97--113.

[36]

Marta Kwiatkowska, Gethin Norman, and David Parker. 2004. Modelling and verification of probabilistic systems. Mathematical Techniques for Analyzing Concurrent and Probabilistic Systems. CRM Monograph Series 23 (2004), 93--215.

[37]

Marta Kwiatkowska, Gethin Norman, and David Parker. 2011. PRISM 4.0: Verification of probabilistic real-time systems. In Computer Aided Verification (LNCS), Ganesh Gopalakrishnan and Shaz Qadeer (Eds.), Vol. 6806. Springer, Berlin, Heidelberg, 585--591.

[38]

Siqi Liu, Adam Wright, and Milos Hauskrecht. 2018. Change-point detection method for clinical decision support system rule monitoring. Artificial Intelligence in Medicine 91 (2018), 49 -- 56.

[39]

Colin Paterson and Radu Calinescu. 2020. Observation-Enhanced QoS Analysis of Component-Based Systems. IEEE Transactions on Software Engineering 46, 5 (2020), 526--548.

[40]

Koushik Sen, Mahesh Viswanathan, and Gul Agha. 2006. Model-checking Markov chains in the presence of uncertainties. In Tools and Algorithms for the Construction and Analysis of Systems (LNCS), Holger Hermanns and Jens Palsberg (Eds.), Vol. 3920. Springer, Berlin, Heidelberg, 394--410.

Digital Library

[41]

Stepan Shevtsov and Danny Weyns. 2016. Keep It SIMPLEX: Satisfying multiple goals with guarantees in control-based self-adaptive systems. In Proc. of the 2016 24th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering (FSE 2016). ACM, New York, NY, USA, 229--241.

Digital Library

[42]

Charles Truong, Laurent Oudre, and Nicolas Vayatis. 2020. Selective review of offline change point detection methods. Signal Processing 167 (2020).

Digital Library

[43]

Gero Walter and Thomas Augustin. 2009. Imprecision and prior-data conflict in generalized Bayesian inference. Journal of Statistical Theory and Practice 3, 1 (2009), 255--271.

[44]

S Zacks. 1983. Survey of classical and Bayesian approaches to the change-point problem: fixed sample and sequential procedures of testing and estimation. In Recent advances in statistics. Elsevier, 245--269.

[45]

Xingyu Zhao, Valentin Robu, David Flynn, Fateme Dinmohammadi, Michael Fisher, and Matt Webster. 2019. Probabilistic model checking of robots deployed in extreme environments. In Proc. of the 33rd AAAI Conference on Artificial Intelligence, Vol. 33. Honolulu, Hawaii, USA, 8076--8084.

Digital Library

[46]

Xiaoqian Zhu, Yongjia Xie, Jianping Li, and Dengsheng Wu. 2015. Change point detection for subprime crisis in American banking: From the perspective of risk dependence. International Review of Economics & Finance 38 (2015), 18 -- 28.

Cited By

Zhao XGerasimou SCalinescu RImrie CRobu VFlynn D(2024)Bayesian learning for the robust verification of autonomous robotsCommunications Engineering10.1038/s44172-024-00162-y3:1Online publication date: 27-Jan-2024
https://doi.org/10.1038/s44172-024-00162-y
Jansen N(2023)Intelligent and Dependable Decision-Making Under UncertaintyFormal Methods10.1007/978-3-031-27481-7_3(26-36)Online publication date: 3-Mar-2023
https://doi.org/10.1007/978-3-031-27481-7_3
Dong YHuang WBharti VCox VBanks AWang SZhao XSchewe SHuang X(2022)Reliability Assessment and Safety Arguments for Machine Learning Components in System AssuranceACM Transactions on Embedded Computing Systems10.1145/357091822:3(1-48)Online publication date: 17-Nov-2022
https://dl.acm.org/doi/10.1145/3570918
Show More Cited By

Index Terms

Interval change-point detection for runtime probabilistic model checking

Recommendations

Probabilistic Model Checking of Biological Systems with Uncertain Kinetic Rates
RP '09: Proceedings of the 3rd International Workshop on Reachability Problems

We present an abstraction of the probabilistic semantics of Multiset Rewriting to formally express systems of reactions with uncertain kinetic rates. This allows biological systems modelling when the exact rates are not known, but are supposed to lie in ...
Probabilistic model checking of biological systems with uncertain kinetic rates

In this paper, we present a formalization of biological systems based on multiset rewriting and we investigate the use of abstract interpretation on its semantics. We consider a probabilistic semantics, which is well suited to represent the non-...
Expression caching for runtime verification based on parameterized probabilistic models
Highlights
- Expansion of efficient runtime verification mechanism by using caching mechanism.
Abstract
Self-adaptive software systems change their behaviors to adapt to their environmental changes at runtime. Runtime verification, which checks the correctness of behaviors after adaptation, sometimes uses probabilistic model checking, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

December 2020

1449 pages

ISBN:9781450367684

DOI:10.1145/3324884

General Chair:
John Grundy,
Program Chairs:
Claire Le Goues,
David Lo

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Engineering and Physical Sciences Research Council

Conference

ASE '20

Sponsor:

ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering

December 21 - 25, 2020

Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
130
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao XGerasimou SCalinescu RImrie CRobu VFlynn D(2024)Bayesian learning for the robust verification of autonomous robotsCommunications Engineering10.1038/s44172-024-00162-y3:1Online publication date: 27-Jan-2024
https://doi.org/10.1038/s44172-024-00162-y
Jansen N(2023)Intelligent and Dependable Decision-Making Under UncertaintyFormal Methods10.1007/978-3-031-27481-7_3(26-36)Online publication date: 3-Mar-2023
https://doi.org/10.1007/978-3-031-27481-7_3
Dong YHuang WBharti VCox VBanks AWang SZhao XSchewe SHuang X(2022)Reliability Assessment and Safety Arguments for Machine Learning Components in System AssuranceACM Transactions on Embedded Computing Systems10.1145/357091822:3(1-48)Online publication date: 17-Nov-2022
https://dl.acm.org/doi/10.1145/3570918
Fang XCalinescu RPaterson CWilson JSchmerl BMaggio MCámara J(2022)PRESTOProceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems10.1145/3524844.3528059(91-97)Online publication date: 18-May-2022
https://dl.acm.org/doi/10.1145/3524844.3528059
Laurent TArcaini PTrubiani CVentresque A(2022)Mutation-based analysis of queueing network performance modelsJournal of Systems and Software10.1016/j.jss.2022.111385191:COnline publication date: 22-Jun-2022
https://dl.acm.org/doi/10.1016/j.jss.2022.111385

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents