article

Free access

Performance Evaluation and Monitoring

Author:

Henry Lucas, Jr.Authors Info & Claims

ACM Computing Surveys (CSUR), Volume 3, Issue 3

Pages 79 - 91

https://doi.org/10.1145/356589.356590

Published: 01 September 1971 Publication History

Abstract

Three major purposes for evaluating the hardware and software performance of computer systems--selection evaluation, performance projection, and performance monitoring--are described. Eight techniques that have been used or suggested for evaluating performance are discussed. Each of these techniques is rated on its suitability for the three purposes of evaluation. Recommendations are made on the most appropriate technique for each evaluation purpose. These suggestions include the development of a comprehensive set of synthetic programs on an industry-wide basis for selection evaluation purposes. Simulation is recommended as the most suitable technique for performance projection. Finally, a number of hardware and software monitors are available for performance monitoring.

References

[1]

ANACKER, W.; AND C. P. WANG. "Performance evaluation of computing systems with memory hierarchies." IEEE Trans. Computers EC-16, 6 (Dec. 1967), 764-773.

[2]

APPLE, C. W. "The program monitor--a device for program performance measurement." Proc. ACM 20th Natl. Conf., 1965, pp. 66-75.

[3]

ARBUCKLE, R. A. "Computer analysis and thruput evaluation." Computers and Automation 15, 1 (Jan. 1966), 12-15, 19.

[4]

BAGLEY, P. R. "Two think pieces." Comm. ACM 3, 1 (Jan. 1960), 1.

[5]

BALDWIN, F. R.; W. B. GIBSON; AND C. B. PO- LAND. "A multiprocessing approach to a large computer system." IBM Systems J. 1, 1 (Sept. 1962), 64-76.

[6]

BEMER, R. W.; AND A. L. ELLISON. "Software instrumentation systems for optimum performance." Proc. IFIP Congress 1968, Vol. 1, North- Holland Publ. Co., Amsterdam, pp. 520-524.

[7]

BONNER, A .J . "Using system monitor output to improve performance." IBM Systems J. 8, 4 (1969), 290-298.

[8]

BUCHOLZ, W. "A synthetic job for measuring system performance." IBM Systems J. 8, 4 (1969), 309-318.

[9]

___, "A selected bibliography on computer system performance evaluation." Computer Group News (March 1969), 21-22.

[10]

BUDD, A.E. "A method for the evaluation of software: general description including benchmark considerations." Mitre Corp. Tech. Rep. 197, vol. 1, 1966.

[11]

___, "A method for the evaluation of softwure: procedural language compilers--particularly COBOL and FORTRAN." Mitre Corp. Tech. Rep. 197, vol. 2, 1966.

[12]

___, "A method for the evaluation of software: executive, operating and monitor systems." Mitre Corp. Tech. Rep. 197, vol. 3, 1966.

[13]

CALINGAERT, P. "System performance evaluation: survey and appraisal." Comm. ACM 10, 1 (Jan. 1967), 12-18.

[14]

CAMPBELL, D. J.; AND W. J. HEFFNER. "Measurement and analysis of large operating systems during system development." Proc. AFIPS 1968 FJCC, Vol. 33, Pt. 1, AFIPS Press, Montvale, N.J., pp. 903-914.

[15]

CANTRELL, H. N. ; AND A. C. ELLISON. "Multiprogramming system performance and analysis." Proc. AF1PS 195'8 SJCC, Vol. 32, AFIPS Press, Montvale, N.J., pp. 213-221.

[16]

CHENG, P.S. "Trace-driven system modeling." IBM Systems J. 8, 4 (1959), 250-289.

[17]

DOPPING, O. "Test problems used for evaluation of computers." BIT 2, 4 (1962). 197-202.

[18]

DRUMMOND, M.E. "A perspective on systems performance evaluation." IBM Systems J. 8, 4 (1969), 252-263.

[19]

GLINKA, L. R.; R. M. BRUSH; AND A. J. UNGAa. "Design, thru simulation, of a multiple-access information system." Proc. AFIPS 1967 FJCC, Vol. 31, AFIPS Press, Montvale, N.J., pp. 437- 447.

[20]

HERMAN, DONALD J. "ScERT: a computer evaluation tool." Datamation 13, 2 (Feb. 1967), 26-28.

[21]

HERMAN, D. J.; AND F. C. IHREa. "The use of a computer to evaluate computers." Proc. AFIPS 1964 SJCC, Vol. 25, Spartan Books, New York, pp. 383-395.

[22]

HUESMANN, L. R.; AND ROBERT P. GOLDBERG. "Evaluating computer systems through simulation." Computer J. 10, 2 (Aug. 1967), 150-155.

[23]

HUTCHINSON, G. K. "A computer center simulation project." Comm. ACM 8, 9 (Sept. 1965), 559-568.

[24]

___, AND J. N. MAGUIRE. "Computer systems design and analysis through simulation." Proc. AF1PS 1965 FJCC, Vol. 27, Pt. 1, Spartan Books, New York, pp. 161-167.

[25]

IHRER, F. C. "Computer performance projected through simulation." Computers and Automation 16, 4 (April 1967), 22-27.

[26]

JOSLIN, E. O. "Applications benchmarks: the key to meaningful computer evaluations." Proc. ACM 2Oth Natl. Con}. 1965, pp. 27-37.

[27]

___, "Cost-value technique for evaluation of computer system proposals." Proc. AFIPS 1964 SJCC, Vol. 25, Spartan Books, New York, pp. 367-381.

[28]

___, AND JOHN J. AIKEN. "The validity of basing computer selection on benchmark results." Computers and Automation 1S, 1 (Jan. 1966), 22-23.

[29]

KATZ, J. H. "Simulation of a multiprocessor computer system." Proc. AFIPS 1966 SJCC, Vol. 28, Spartan Books, New York, pp. 127-139.

[30]

. "An experimental model of the System/ 360." Comm. ACM 10, 11 (Nov. 1967), 694-702.

[31]

KEEFE, D. D. "Hierarchical control programs for systems evaluation." IBM Systems J. 7, 2 (1968), 123-133.

[32]

KNIGHT, K. E. "Changes in computer performance." Datamatiou 12, 9 (Sept. 1966), 40-- 54.

[33]

___, "Evolving computer performance 1963- 1967." Datamation 14, 6 (Jan. 1968), 31-35.

[34]

LEHMAN, M. M.; AND J. L. ROSENFELD. "Performance of a simulated multi-programming system." Proc. AFIPS 1968 FJCC, Vol. 33, Pt. 2, AFIPS Press, Montvale, N.J., pp. 1431- 1442.

[35]

Lucas, HENRY C., JR. "Computer evaluation through benchmark testing." Unpublished paper, MIT, Cambridge, Mass., May, 1967.

[36]

MERIKALLIO, R. A.; AND F. C. HOLLAND. "Simulation design of a multiprocessing system." Proc. AFIPS 1968 FJCC, Vol. 33, Pt. 2, AFIPS Press, Montvale, N.J., pp. 1399-1410.

[37]

MURPHY, JESSE O.; AND ROBERT M. WADE. "The IBM 360/195 in a world of mixed job streams." Datamation 16, 4 (April 1970), 72- 79.

[38]

NIELSEN, NORMAN R. "The simulation of timesharing systems." Comm. ACM 10, 7 (July 1967), 397-412.

[39]

___, "Computer simulation of computer system performance." Proc. ACM 22nd Natl. Conj., 1967, MDI PubIns., Wayne, Pa., pp. 581-590.

[40]

OPLER, ASCHER. "Measurement of software characteristics." Datamation 10, 7 (July 1964), 27-30.

[41]

PATRICK, ROBERT L. "Measuring performance." Datamation 10, 7 (July 1964), 24-27.

[42]

RAICHELSON, E.; AND G. COLLINS. "A method for comparing the internal operating speeds of computers." Comm. ACM 7, 5 (May 1964), 309-310.

[43]

ROSIN, R. F. "Determining a computing center environment." Comm. ACM 8, 7 (July 1965), 463-468.

[44]

SCHERR, ALLAN L, "Time-sharing measurement." Datamation 12, 4 (April 1956), 22-26.

[45]

SCHULMAN, F. D. "Hardware measurement device for IBM System/360 time-sharing evaluation." Proc. ACM 22rid Natl. Conf., 1967, MDI PubIns., Wayne, Pa., pp. 103-109.

[46]

SCHWARTZ, EUGENE S. "Computer evaluation and selection." J. Data Management 6, 6 (June 1968), 58--62.

[47]

SEAMAN, P. H.; AND R. C. SoucY. "Simulating operating systems." IBM Systems J. 8, 4 (1969), 264-279.

[48]

SHARPB, WILLIAM F. The economics of computers. Columbia University Press, New York, 1969.

[49]

SMITH, J. L. "An analysis of time-sharing computer systems using Markov models." Proc. AFIPS 1966 SJCC, Vol. 28, Spartan Books, New York, pp. 87-95.

[50]

STANG, HAROLD; AND PETER SOUTHGATE. "Performance evaluation of third-generation computing systems." Datamation 15, 11 (Nov. 1969), 181-190.

[51]

STANLEY, W. I. "Measurement of system operational statistics." IBM Systems J. 8, 4 (1969), 299-3O8.

[52]

___, AND H. F. HERTEL. "Statistics gathering and simulation for the Apollo real-time operating system." IBM Systems J. 7, 2 (1968), 85-102.

[53]

STATLAND, NOaMAN. "Methods of evaluating computer systems performance." Computers and Automation 13, 2 (Feb. 1964), 18--23.

[54]

STIMLER, S.; AND K. A. BRONS. "A methodology for calculating and optimizing real-time system performance." Comm. ACM 11, 7 (July 1968), 509-516.

[55]

TOTARO, J. B. "Real-time processing power: a standardized evaluation." Computers and Automation 17, 4 (April 1967), 16-19.

[56]

WALTER, E. S. ; AND V. L. WALLACE. "Further analysis of a computing center environment." Comm. ACM 10, 5 (May 1967), 266-272.

[57]

WHITE, PETER. "Relative effects of central processor and input/output speeds upon throughput on the large computer." Comm. ACM 7, 12 (Dec. 1964), 711-714.

[58]

WILLIAMS, 0.; ET AL. "A methodology for computer selection studies." Computers and Automation 12, 5 (May 1963), 18--23.

[59]

YOUCHAH, M. I.; ET AL. "The data processing system simulator (DPSS)." Proc. AFIPS 1964 FJCC, Vol. 26, Pt. 1, Spartan Books, New York, pp. 251-276.

Cited By

Muehlburger HWotawa F(2024)Making Systems Fail-Aware: A Semi-Supervised Machine Learning Approach for Identifying Failures by Learning the Correct Behavior of a SystemIFAC-PapersOnLine10.1016/j.ifacol.2024.07.18558:4(7-12)Online publication date: 2024
https://doi.org/10.1016/j.ifacol.2024.07.185
Peñaflor J(2024)Clicks, Queries, and Searches: Uncovering User Behavior Through Transaction Log AnalysisInformation Experience and Information Literacy10.1007/978-3-031-53001-2_17(186-198)Online publication date: 1-Feb-2024
https://doi.org/10.1007/978-3-031-53001-2_17
Terrosi FMariotti FLollini PBondavalli A(2023)Modeling of GPGPU architectures for performance analysis of CUDA programs2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00079(761-771)Online publication date: 22-Oct-2023
https://doi.org/10.1109/QRS60937.2023.00079
Show More Cited By

Recommendations

Promoting the Public Service Performance Evaluation: An Alternative View from Performance Culture Perspective
BIFE '12: Proceedings of the 2012 Fifth International Conference on Business Intelligence and Financial Engineering

The evaluation of public service performance plays a significant part in government's performance management system. It is the mechanism of satisfying the public demands and constructing the service-oriented society. However, at present there are still ...
System of CRM Performance Evaluation Based on Fuzzy Comprehensive Algorithm
ICIII '08: Proceedings of the 2008 International Conference on Information Management, Innovation Management and Industrial Engineering - Volume 01

With the level of informatization is improving CRM has been used by enterprises universally and it makes the working efficiency has been improved. In order to improve the CRM performance needs to do a scientific and comprehensive evaluation. The ...
Issues in Performance Evaluation: A Case Study of Math Recognition
ICDAR '09: Proceedings of the 2009 10th International Conference on Document Analysis and Recognition

Performance evaluation of document recognition systems is a difficult and practically important problem. Issues arise in defining requirements, in characterizing the system’s range of inputs and outputs, in interpreting published performance evaluation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 3, Issue 3

Sept. 1971

67 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/356589

Issue’s Table of Contents

Copyright © 1971 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 September 1971

Published in CSUR Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

145
Total Citations
View Citations
6,959
Total Downloads

Downloads (Last 12 months)1,890
Downloads (Last 6 weeks)193

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Muehlburger HWotawa F(2024)Making Systems Fail-Aware: A Semi-Supervised Machine Learning Approach for Identifying Failures by Learning the Correct Behavior of a SystemIFAC-PapersOnLine10.1016/j.ifacol.2024.07.18558:4(7-12)Online publication date: 2024
https://doi.org/10.1016/j.ifacol.2024.07.185
Peñaflor J(2024)Clicks, Queries, and Searches: Uncovering User Behavior Through Transaction Log AnalysisInformation Experience and Information Literacy10.1007/978-3-031-53001-2_17(186-198)Online publication date: 1-Feb-2024
https://doi.org/10.1007/978-3-031-53001-2_17
Terrosi FMariotti FLollini PBondavalli A(2023)Modeling of GPGPU architectures for performance analysis of CUDA programs2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00079(761-771)Online publication date: 22-Oct-2023
https://doi.org/10.1109/QRS60937.2023.00079
Volpert SEichhammer PHeld FHuffert TReiser HDomaschka J(2023)The view on systems monitoring and its requirements from future Cloud-to-Thing applications and infrastructuresFuture Generation Computer Systems10.1016/j.future.2022.11.024141:C(243-257)Online publication date: 15-Feb-2023
https://dl.acm.org/doi/10.1016/j.future.2022.11.024
Basavegowda Ramu VYeruva A(2023)The Capability of Observing Performance in Healthcare SystemsComputational Intelligence for Clinical Diagnosis10.1007/978-3-031-23683-9_39(541-548)Online publication date: 6-Jun-2023
https://doi.org/10.1007/978-3-031-23683-9_39
AL-Saadi TCherepovitsyn ASemenova T(2022)Iraq Oil Industry Infrastructure Development in the Conditions of the Global Economy TurbulenceEnergies10.3390/en1517623915:17(6239)Online publication date: 26-Aug-2022
https://doi.org/10.3390/en15176239
Semenova TAl-Dirawi AAl-Saadi T(2022)Environmental Challenges for Fragile Economies: Adaptation Opportunities on the Examples of the Arctic and IraqAgronomy10.3390/agronomy1209202112:9(2021)Online publication date: 26-Aug-2022
https://doi.org/10.3390/agronomy12092021
Zhang KXie YNoorkhah SImeni MDas S(2022)Neutrosophic management evaluation of insurance companies by a hybrid TODIM-BSC method: a case study in private insurance companiesManagement Decision10.1108/MD-01-2022-012061:2(363-381)Online publication date: 12-Apr-2022
https://doi.org/10.1108/MD-01-2022-0120
García-Domínguez APalomo-Lozano FMedina-Bulo IIbias ANúñez M(2022)Computing performance requirements for web service compositionsComputer Standards & Interfaces10.1016/j.csi.2022.10366483:COnline publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1016/j.csi.2022.103664
Raducu RRodríguez RÁlvarez P(2022)Resource Consumption Evaluation of C++ Cryptographic Libraries on Resource-Constrained DevicesApplied Cryptography in Computer and Communications10.1007/978-3-031-17081-2_5(65-75)Online publication date: 6-Oct-2022
https://doi.org/10.1007/978-3-031-17081-2_5
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents