research-article

A Framework for Computer Performance Evaluation Using Benchmark Sets

Authors:

Umesh Krishnaswamy,

Isaac D. SchersonAuthors Info & Claims

IEEE Transactions on Computers, Volume 49, Issue 12

Pages 1325 - 1338

https://doi.org/10.1109/12.895853

Published: 01 December 2000 Publication History

Abstract

Benchmarking is a widely used approach to measure computer performance. Current use of benchmarks only provides running times to describe the performance of a tested system. Glancing through these execution times provides little or no information about system strengths and weaknesses. A novel benchmarking methodology is proposed to identify key performance parameters; the methodology is based on measuring performance vectors. A performance vector is a vector of ratings that represents delivered performance of primitive operations of a system. In order to measure performance vectors, a geometric model is proposed which defines system behavior using the concepts of support points, context lattice, and operating points. In addition to the performance vector, other metrics derivable from the geometric model include the variation in system performance and the compliance of benchmarks. Using this methodology, the performance vectors of the Sun SuperSPARC (desktop workstation) and the Cray C90 (vector supercomputer) are evaluated using the SPEC benchmarks and the Perfect Club, respectively. The proposed methodology respects several practical constraints and issues in benchmarking. The instrumentation required is minimal. The benchmarks used are realistic (not synthetic) in order to reflect the delivered (not peak) performance. Finally, operations in the performance vector are not measured individually since there may be significant interplay in their executions.

References

[1]

A.K. Agrawala R.M. Bryant and J.M. Mohr, “An Approach to the Workload Characterization Problem,” Computer, vol. 9, pp. 18-32, 1976.

Digital Library

[2]

D. Bailey E. Barszcz L. Dagum and H.D. Simon, “The NAS Parallel Benchmarks Results 10-94,” Technical Report NAS-94-001, NAS Systems Division, NASA Ames Research Center, Oct. 1994.

[3]

D. Bailey J. Barton T. Lasinski and H. Simon, “The NAS Parallel Benchmarks,” Technical Report RNR-91-002 Revision 2, NAS Systems Division, NASA Ames Research Center, Aug. 1991.

[4]

T. Ball and J.L. Larus, “Optimally Profiling and Tracing Programs,” Technical Report 1031 (Revision 1), Computer Science Dept., Univ. of Wisconsin, Madision, Sept. 1991.

[5]

Y. Bard, “Performance Criteria and Measurement for a Time-Sharing System,” IBM Systems J., vol. 10, no. 3, pp. 193-216, 1971.

Digital Library

[6]

Y. Bard and K.V. Suryanarayana, “On the Structure of CP-67 Overhead,” Statistical Computer Performance Evaluation, W. Freiberger, ed., pp. 329-346, New York: Academic Press, 1972.

[7]

M. Berry D. Chen P. Koss D. Kuck S. Lo Y. Pang L. Pointer R. Roloff A. Sameh E. Clementi S. Chin D. Schneider G. Fox P. Messina D. Walker C. Hsuing J. Schwarzmeier K. Lue S. Orzag F. Seidl O. Johnson R. Goodrum and J. Martin, “The Perfect Club Benchmarks: Effective Performance Evaluation of Supercomputers,” Int'l J. Supercomputing Applications, vol. 3, no. 3, pp. 5-40, 1989.

Digital Library

[8]

M. Berry, et al., “The Perfect Club Benchmarks: Effective Performance Evaluation of Supercomputers,” Int'l J. Supercomputing Applications, vol. 3, no. 3, pp. 5-40, 1989.

Digital Library

[9]

M. Calzarossa and G. Serazzi, “Workload Characterization: A Survey,” Proc. IEEE, vol. 81, no. 8, pp. 1,136-1,150, Aug. 1993.

[10]

R. F. Cmelik and D. Keppel, “Shade: A Fast Instruction-Set Simulator for Execution Profiling,” Proc. 1994 ACM SIGMETRICS Conf. Measurement and Modeling of Computer Systems, pp. 128-137, May 1994.

Digital Library

[11]

CSRD Staff, “Perfect Report 2: Addendum 1,” Technical Report CSRD Report 1052, Center for Supercomputing Research and Development, Univ. of Illinois at Urbana-Champaign, Feb. 1991.

[12]

CSRD Staff, “Perfect Report 2: Addendum 2,” Technical Report CSRD Report 1168, Center for Supercomputing Research and Development, Univ. of Illinois at Urbana-Champaign, Nov. 1991.

[13]

U. Detert and G. Hofemann, “Cray X-MP and Y-MP Memory Performance,” Parallel Computing, vol. 17, nos. 4-5, pp. 579-590, July 1991.

Digital Library

[14]

K.M. Dixit, “The SPEC Benchmarks,” Parallel Computing, vol. 17, nos. 10-11, pp. 1,195-1,210, Dec. 1991.

Digital Library

[15]

J.J. Dongarra, “Performance of Various Computers Using Standard Linear Equations Software,” Technical Report CS-89-85, Computer Science Dept., Univ. of Tennessee, Knoxville, 1989.

Digital Library

[16]

A.V. Fiacco G.P. McCormick, Nonlinear Programming: Sequential Unconstrained Minimization Techniques. New York: John Wiley and Sons, 1968.

[17]

L. Geppert, “Not Your Father's CPU,” IEEE Spectrum, vol. 30, no. 12, pp. 20-23, Dec. 1993.

Digital Library

[18]

P. Heidelberger and S. Lavenberg, “Computer Performance Evaluation Methodology,” IEEE Trans. Computers, vol. 33, no. 12, pp. 1,195-1,220, Dec. 1984.

[19]

J.L. Hennessy and D.A. Patterson, Computer Architecture—A Quantitative Approach, second ed. San Francisco: Morgan Kaufmann, 1996.

Digital Library

[20]

P.Y.-T. Hsu, Introduction to SHADOW, Revision A. Mountain View, Calif.: Sun Microsystems, Inc., July 1989.

[21]

A. Inoue and K. Takeda, “Performance Evaluation for Various Configurations of Superscalar Processors,” Computer Architecture News, vol. 21, no. 1, pp. 4-11, Mar. 1993.

Digital Library

[22]

R.A. Kamin III G.B. Adams III and P.K. Dubey, “Dynamic Trace Analysis for Analytical Modeling of Superscalar Performance,” Performance Evaluation, vol. 19, nos. 2-3, pp. 259-276, Mar. 1994.

Digital Library

[23]

U. Krishnaswamy, “Computer Evaluations Using Performance Vectors,” technical report, Dept. of Information and Computer Science, Univ. of California, Irvine, Dec. 1995.

[24]

U. Krishnaswamy and I.D. Scherson, “Micro-Architecture Evaluation Using Performance Vectors,” Proc. ACM Sigmetrics Conf. Measurement and Modeling of Computer Systems, pp. 148-159, May 1996.

Digital Library

[25]

T.T. Kwan B.K. Totty and D.A. Reed, “Communication and Computation Performance of the CM-5,” Proc. Supercomputing '93, pp. 192-201, Nov. 1993.

Digital Library

[26]

C.L. Lawson and R.J. Hanson, Solving Least Squares Problems. Englewood Cliffs, N.J.: Prentice Hall, 1974.

[27]

R.L. Lee A.Y. Kwok and F.A. Briggs, “The Floating Point Performance of a Superscalar SPARC Processor,” SIGPLAN Notices, vol. 26, no. 4, pp. 28-37, Apr. 1991.

Digital Library

[28]

T. Manley and H. Grossman, “Window Overflow Reduction for SPARC Processors,” Proc. 31st Ann. Southeast Conf., pp. 56-64, 1994.

[29]

L. McMahan and R. Lee, “Pathlengths of SPEC Benchmarks for PA-RISC, MIPS, and SPARC,” Digest of Papers COMPCON Spring '93, pp. 481-490, Feb. 1993.

[30]

MIPS Computer Systems, Inc., MIPS Languages and Programmers's Manual, 1986.

[31]

A. Nanda and L.M. Ni, “Benchmark Workload Generation and Performance Characterization of Multiprocessors,” Proc. Supercomputing '92, pp. 20-29, Nov. 1992.

Digital Library

[32]

D.B. Noonburg and J.P. Shen, “Theoretical Modeling of Superscalar Processor Performance,” Proc. 27th Ann. Int'l Symp. Microarchitecture MICRO 27, pp. 52-62, Dec. 1994.

Digital Library

[33]

R.W. Numrich P.L. Springer and J.C. Peterson, “Measurement of Communication Rates on the Cray T3D Interprocessor Network,” High-Performance Computing and Networking, W. Gentzsch and U. Harms, eds., pp. 150-157, Berlin: Springer-Verlag, 1994.

Digital Library

[34]

W. Oed, “Y-MP C90: System Features and Early Benchmark Results,” Parallel Computing, vol. 18, no. 8, pp. 947-954, Aug. 1992.

Digital Library

[35]

W. Oed, personal communications, Cray Research, GmbH, München, Germany, 1995.

[36]

L. Pointer, “Perfect Report 2,” Technical Report CSRD Report 964, Center for Supercomputing Research and Development, Univ. of Illinois at Urbana-Champaign, Mar. 1990.

[37]

D.A. Reed R.A. Aydt R.J. Noe P.C. Roth K.A. Shields B.W. Schwartz and L.F. Tavera, “Scalable Performance Analysis: The Pablo Performance Analysis Environment,” Proc. Scalable Parallel Libraries Conf., pp. 104-113, Oct. 1993.

[38]

K.A. Robbins and S. Robbins, “Dynamic Behavior of Memory Reference Streams for the Perfect Club Benchmarks,” Proc. Int'l Conf. Parallel Processing, pp. I-48-52, 1992.

[39]

R.H. Saavedra-Barrera and A.J. Smith, “Analysis of Benchmark Characteristics and Benchmark Performance Prediction,” Technical Report USC-CS-92-524, Univ. of Southern California, Los Angeles, Sept. 1992.

[40]

R.H. Saavedra-Barrera A.J. Smith and E. Miya, “Machine Characterization Based on an Abstract High-Level Language Machine,” IEEE Trans. Computers, vol. 38, no. 12, pp. 1,659-1,679, Dec. 1989.

Digital Library

[41]

M.J. Serrano W. Yamamoto R.C. Wood and M. Nemirovsky, “A Model for Performance Estimation in a Multistreamed Superscalar Processor,” Computer Performance Evaluation: Modelling Techniques and Tools, G. Haring and G. Kotsis, eds., pp. 213-230, Berlin: Springer-Verlag, 1994.

Digital Library

[42]

P. Sinvhal-Sharma, “Perfect Benchmarks™ Documentation Suite 1,” Center for Supercomputing Research and Development, Univ. of Illinois, Urbana-Champaign, Sept. 1991.

[43]

SPEC, SPEC Newsletter, June 1994.

[44]

G.W. Stewart, Introduction to Matrix Computations. New York: Academic Press, 1973.

[45]

G. Strang, Linear Algebra and Its Applications, third ed. San Diego, Calif.: Harcourt Brace Jovanovich, 1988.

[46]

Sun Microsystems, Inc., The SuperSPARC User's Guide, Part No. 801-4272-01, year?

[47]

Sun Microsystems, Inc., The SPARC Architecture Manual, Version 8, Part No. 800-1399-09, Aug. 1989.

[48]

D. Tabak, Advanced Microprocessors, second ed. New York: McGraw-Hill, 1885.

Digital Library

[49]

S. Vajapeyam G.S. Sohi and W.-C. Hsu, “An Empirical Study of the Cray Y-MP Using the Perfect Club Benchmarks,” Proc. 18th Int'l Symp. Computer Architecture, pp. 170-179, 1991.

Digital Library

[50]

S. Wallace and N. Bagherzadeh, “Performance Issues of a Superscalar Microprocessor,” Microprocessors and Microsystems, vol. 19, no. 4, pp. 187-199, May 1995.

Cited By

Wang BSong YSun YLiu J(2019)Analysis model for server consolidation of virtualized heterogeneous data centers providing internet servicesCluster Computing10.1007/s10586-018-2880-x22:3(911-928)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s10586-018-2880-x
Rolia JKalbasi AKrishnamurthy DDawson SAdamson ABondi AJuiz CSquillante M(2010)Resource demand modeling for multi-tier servicesProceedings of the first joint WOSP/SIPEW international conference on Performance engineering10.1145/1712605.1712638(207-216)Online publication date: 28-Jan-2010
https://dl.acm.org/doi/10.1145/1712605.1712638
Rolia JKrishnamurthy DCasale GDawson SAdamson ABondi AJuiz CSquillante M(2010)BAPProceedings of the first joint WOSP/SIPEW international conference on Performance engineering10.1145/1712605.1712609(3-14)Online publication date: 28-Jan-2010
https://dl.acm.org/doi/10.1145/1712605.1712609
Show More Cited By

Index Terms

A Framework for Computer Performance Evaluation Using Benchmark Sets

Recommendations

Computer evaluation using performance vectors
Micro-architecture evaluation using performance vectors
SIGMETRICS '96: Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems

Benchmarking is a widely used approach to measure computer performance. Current use of benchmarks only provides running times to describe the performance of a tested system. Glancing through these execution times provides little or no information about ...
Analysis of Benchmark Characteristics and Benchmark Performance

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Computers

IEEE Transactions on Computers Volume 49, Issue 12

December 2000

105 pages

ISSN:0018-9340

Editor:
Jean-Luc Gaudiot
Univ. of Southern California

Issue’s Table of Contents

Copyright © Copyright © 2000 IEEE. All Rights Reserved.

Publisher

IEEE Computer Society

United States

Publication History

Published: 01 December 2000

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang BSong YSun YLiu J(2019)Analysis model for server consolidation of virtualized heterogeneous data centers providing internet servicesCluster Computing10.1007/s10586-018-2880-x22:3(911-928)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1007/s10586-018-2880-x
Rolia JKalbasi AKrishnamurthy DDawson SAdamson ABondi AJuiz CSquillante M(2010)Resource demand modeling for multi-tier servicesProceedings of the first joint WOSP/SIPEW international conference on Performance engineering10.1145/1712605.1712638(207-216)Online publication date: 28-Jan-2010
https://dl.acm.org/doi/10.1145/1712605.1712638
Rolia JKrishnamurthy DCasale GDawson SAdamson ABondi AJuiz CSquillante M(2010)BAPProceedings of the first joint WOSP/SIPEW international conference on Performance engineering10.1145/1712605.1712609(3-14)Online publication date: 28-Jan-2010
https://dl.acm.org/doi/10.1145/1712605.1712609
Casale GKalbasi AKrishnamurthy DRolia J(2009)Automatic stress testing of multi-tier systems by dynamic bottleneck switch generationProceedings of the ACM/IFIP/USENIX 10th international conference on Middleware10.5555/1813355.1813383(393-413)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1813355.1813383
Casale GKalbasi AKrishnamurthy DRolia JCampbell RBacon JCooper B(2009)Automatic stress testing of multi-tier systems by dynamic bottleneck switch generationProceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware10.5555/1656980.1657008(1-20)Online publication date: 30-Nov-2009
https://dl.acm.org/doi/10.5555/1656980.1657008
Bataineh S(2005)Toward an analytical solution to task allocation, processor assignment, and performance evaluation of network processorsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2004.09.00865:1(29-47)Online publication date: 1-Jan-2005
https://dl.acm.org/doi/10.1016/j.jpdc.2004.09.008
Soininen JJantsch AForsell MPelkonen AKreku JKumar S(2003)Extending Platform-Based Design to Network on Chip SystemsProceedings of the 16th International Conference on VLSI Design10.5555/832285.835539Online publication date: 4-Jan-2003
https://dl.acm.org/doi/10.5555/832285.835539
Bjuréus PJantsch AHermida RAboulhamid E(2001)Performance analysis with confidence intervals for embedded software processesProceedings of the 14th international symposium on Systems synthesis10.1145/500001.500012(45-50)Online publication date: 30-Sep-2001
https://dl.acm.org/doi/10.1145/500001.500012

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents