article

Paper: The SPEC benchmarks

Author:

Kaivalya M. DixitAuthors Info & Claims

Parallel Computing, Volume 17, Issue 10-11

Pages 1195 - 1209

https://doi.org/10.1016/S0167-8191(05)80033-X

Published: 01 December 1991 Publication History

Abstract

This paper characterizes problems with popular benchmarks used to measure system performance and describes the history, structure, mission, future, and workings of an evolving standard in characterizing systems performance. Systems Performance Evaluation Cooperative (SPEC) represents an effort of major computer companies to create a yardstick to measure the performance of computer systems. This paper compares results of three architectures with traditional performance metrics and metrics developed by SPEC. It discusses strengths and weaknesses of SPEC and warns users about the dangers of single number performance characterization.

References

[1]

Bailey, D.H. and Barton, J.T., The NAS Kernel benchmark program. 1985.

[2]

D.M. Betz, XLISP 1.6, Rev. 1.0, November 1988.

[3]

Carr, S. and Kennedy, K., Blocking linear algebra codes for memory hierarchies. In: Fourth Siam Conf. on Parallel Processing for Scientific Computing,

[4]

Conte, T.M. and Hwu, W.M.W., Benchmark characterization. In: IEEE Comput.,

[5]

Curnow, H.J. and Wichman, B.A., A synthetic benchmark. Comput. J.

[6]

Cmelik, R.F., Kong, S.I., Ditzel, D.R. and Kelly, E.J., An analysis of MIPS and SPARC instruction set utilization on the SPEC benchmarks, SMI. 1991.

[7]

Doduc, N., FORTRAN execution time benchmark, Framentec, V29. 1989.

[8]

Dongarra, J.J., Performance of various computers using standard linear equation software. 1989.

[9]

Hennessy, J.L. and Patterson, D.A., Computer Architecture, A Quantitative Approach. 1990.

[10]

Hennessy, J.L. and Patterson, D.A., Computer Architecture, A Quantitative Approach. In: pp. 73-74.

[11]

Hennessy, J.L. and Patterson, D.A., Computer Architecture, A Quantitative Approach. 1990.

[12]

Hill, M.D., Cache performance of the Integer SPEC benchmarks on a RISC. Comput. Architecture News. v18 i2.

[13]

J.R. Mashey, Your mileage may vary***but if your car were a computer, it would vary more, Issue 2.0, 05/18/90 MIPS Computer Systems.

[14]

Portfield, A.K., Software methods for improvement of cache performance on supercomputer applications. In: PhD thesis, Rice University, Los Alamos, NM.

[15]

Savedra-Barrera, R.H., The SPEC and Perfect Club benchmarks: Promises and limitations. In: Hot Chips-2 Symp.,

[16]

Smith, J.E., Characterizing computer performance with a single number. Commun. ACM. 1202-1206.

[17]

Dixit, K.M., Speculations: Defining the SPEC benchmark. SunTech J. v4 i1.

[18]

SPEC Staff, Benchmark result reporting format terminology. SPEC Newsletter. v3 i1.

[19]

Dixit, K. and Novak, R., SPEC refines reporting format to ensure level playing field. SPEC Benchmark Results. v2 i1.

[20]

EECS Department of U.C. Berkeley, Espresso Version #2.3, Release Date 01/31/88.

[21]

CAD/IC Group, EECS/ERL of U.C. Berkeley, SPICE2G.6, Release Date March 1987.

[22]

The Industrial Liaison Program of U.C. Berkeley, Eqntott #V9, Released 1985.

[23]

Weicker, R., Dhrystone: A synthetic systems programming benchmark. Commun. ACM.

[24]

Weicker, R., An overview of common benchmarks. Microprocessor Forum.

[25]

Weicker, R.P., Understanding variations in Dhrystone performance. Microprocessor Report.

Cited By

Tsuji MKramer WWeill JNominé JSato M(2021)A new sustained system performance metric for scientific performance evaluationThe Journal of Supercomputing10.1007/s11227-020-03545-y77:7(6476-6504)Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.1007/s11227-020-03545-y
Rho SPark GChoi JPark C(2021)Development of benchmark automation suite and evaluation of various high-performance computing systemsCluster Computing10.1007/s10586-020-03167-224:1(159-179)Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1007/s10586-020-03167-2
Stratis PYaneva VRajan AOivo MMéndez DMockus A(2018)Assessing the effect of data transformations on test suite compilationProceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3239235.3240499(1-10)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3239235.3240499
Show More Cited By

Index Terms

Paper: The SPEC benchmarks
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Social and professional topics
  1. Professional topics
    1. Computing profession
      1. Testing, certification and licensing

Index terms have been assigned to the content through auto-classification.

Recommendations

How to Build a Benchmark
ICPE '15: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

Standardized benchmarks have become widely accepted tools for the comparison of products and evaluation of methodologies. These benchmarks are created by consortia like SPEC and TPC under confidentiality agreements which provide little opportunity for ...
Paper: A detailed look at some popular benchmarks

'Fair benchmarking' can be less of an oxymoron - if the people using benchmark results know what tasks the benchmarks really perform and what they measure. This paper characterizes in detail the most commonly used industry standard benchmark programs (...
Further implementation aspects of the server efficiency rating tool (SERT)
ICPE '13: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering

The Server Efficiency Rating Tool (SERT) has been developed by the Standard Performance Evaluation Corporation (SPEC) at the request of the US Environmental Protection Agency (EPA). Almost 3% of all electricity consumed within the US in 2010 went to ...

Comments

Information & Contributors

Information

Published In

cover image Parallel Computing

Parallel Computing Volume 17, Issue 10-11

December, 1991

241 pages

ISSN:0167-8191

Issue’s Table of Contents

Copyright © Elsevier Science Publishers B.V. All rights reserved © 1991.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 01 December 1991

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tsuji MKramer WWeill JNominé JSato M(2021)A new sustained system performance metric for scientific performance evaluationThe Journal of Supercomputing10.1007/s11227-020-03545-y77:7(6476-6504)Online publication date: 1-Jul-2021
https://dl.acm.org/doi/10.1007/s11227-020-03545-y
Rho SPark GChoi JPark C(2021)Development of benchmark automation suite and evaluation of various high-performance computing systemsCluster Computing10.1007/s10586-020-03167-224:1(159-179)Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1007/s10586-020-03167-2
Stratis PYaneva VRajan AOivo MMéndez DMockus A(2018)Assessing the effect of data transformations on test suite compilationProceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3239235.3240499(1-10)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3239235.3240499
Jang JYang HYoo SHessel FRousseau FKent KLee K(2017)Executable dataflow benchmark generation technique for multi-core embedded systemsProceedings of the 28th International Symposium on Rapid System Prototyping: Shortening the Path from Specification to Prototype10.1145/3130265.3130323(50-56)Online publication date: 19-Oct-2017
https://dl.acm.org/doi/10.1145/3130265.3130323
Ghasempour MJaleel AGarside JLuján MJacob B(2016)DReAMProceedings of the Second International Symposium on Memory Systems10.1145/2989081.2989102(362-373)Online publication date: 3-Oct-2016
https://dl.acm.org/doi/10.1145/2989081.2989102
Ghasempour MJaleel AGarside JLuján MJacob B(2016)HAPPYProceedings of the Second International Symposium on Memory Systems10.1145/2989081.2989101(311-321)Online publication date: 3-Oct-2016
https://dl.acm.org/doi/10.1145/2989081.2989101
Park JOrshansky MCoskun AMargala MBehjat LHan J(2016)Multiple Attempt Write Strategy for Low Energy STT-RAMProceedings of the 26th edition on Great Lakes Symposium on VLSI10.1145/2902961.2903015(163-168)Online publication date: 18-May-2016
https://dl.acm.org/doi/10.1145/2902961.2903015
Schneider TGerstenberger RHoefler T(2012)Micro-applications for communication data access patterns and MPI datatypesProceedings of the 19th European conference on Recent Advances in the Message Passing Interface10.1007/978-3-642-33518-1_17(121-131)Online publication date: 23-Sep-2012
https://dl.acm.org/doi/10.1007/978-3-642-33518-1_17
Krishnaswamy UScherson I(2000)A Framework for Computer Performance Evaluation Using Benchmark SetsIEEE Transactions on Computers10.1109/12.89585349:12(1325-1338)Online publication date: 1-Dec-2000
https://dl.acm.org/doi/10.1109/12.895853
Hempel RCalkin RHess RJoppich WOosterlee CRitzdorf HWypior PZiegler WKoike NWashio TKeller U(1996)Real applications on the new parallel system NEC Cenju-3Parallel Computing10.1016/0167-8191(95)00060-722:1(131-148)Online publication date: 1-Jan-1996
https://dl.acm.org/doi/10.1016/0167-8191%2895%2900060-7
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents