A statistical approach to diagnosing intermittent performance-problems using monotone relationships

Managing a computer system requires that good performance (e.g., large throughputs, small response times) be maintained in order to meet business objectives. Rarely is performance consistently bad. More frequently, performance is good one day and bad the next. Diagnosing such intermittent performance-problems involves determining what distinguishes bad days from good days, such as larger paging rates. Once this is understood, an appropriate remedy can be found, such as buying more memory. This paper describes a statistical approach to diagnosing intermittent performance-problems when the relationships among measurement variables are expressed qualitatively as monotone relationships (e.g., paging delays increase with the number of logged-on users). We present a non-parametric test for monotonicity (NTM) that evaluates monotone relationships based on F_A, the fraction of observation-pairs that agree with the monotone relationship. An interpretation of F_A in terms of statistical significance levels is presented, and NTM is compared to least-squares regression. Based on NTM, an algorithm for diagnosing intermittent performance-problems is presented. NTM and our diagnosis algorithm are applied to measurements of four similarly configured IBM 9370 model 60s running IBM's operating-system Virtual Machine System Product (VM SP).