research-article

IntroPerf: transparent context-sensitive multi-layer performance inference using system stack traces

Authors:

Chung Hwan Kim,

Dongyan XuAuthors Info & Claims

SIGMETRICS '14: The 2014 ACM international conference on Measurement and modeling of computer systems

Pages 235 - 247

https://doi.org/10.1145/2591971.2592008

Published: 16 June 2014 Publication History

Abstract

Performance bugs are frequently observed in commodity software. While profilers or source code-based tools can be used at development stage where a program is diagnosed in a well-defined environment, many performance bugs survive such a stage and affect production runs. OS kernel-level tracers are commonly used in post-development diagnosis due to their independence from programs and libraries; however, they lack detailed program-specific metrics to reason about performance problems such as function latencies and program contexts. In this paper, we propose a novel performance inference system, called IntroPerf, that generates fine-grained performance information -- like that from application profiling tools -- transparently by leveraging OS tracers that are widely available in most commodity operating systems. With system stack traces as input, IntroPerf enables transparent context-sensitive performance inference, and diagnoses application performance in a multi-layered scope ranging from user functions to the kernel. Evaluated with various performance bugs in multiple open source software projects, IntroPerf automatically ranks potential internal and external root causes of performance bugs with high accuracy without any prior knowledge about or instrumentation on the subject software. Our results show IntroPerf's effectiveness as a lightweight performance introspection tool for post-development diagnosis.

References

[1]

Event Tracing for Windows (ETW). http://msdn.microsoft.com/en-us/library/windows/desktop/aa363668(v=vs.85).aspx.

[2]

Ftrace: Function Tracer. https://www.kernel.org/doc/Documentation/trace/ftrace.txt.

[3]

gperftools: Fast, multi-threaded malloc() and nifty performance analysis tools. https://code.google.com/p/gperftools/.

[4]

jstack - Stack Trace. http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html.

[5]

LTTng: Linux Tracing Toolkit - next generation. http://lttng.org.

[6]

Oprofile: a system-wide profiler for linux systems. http://oprofile.sourceforge.net/.

[7]

perf: Linux profiling with performance counters. https://perf.wiki.kernel.org/.

[8]

Stack Walking (Windows Driver). http://msdn.microsoft.com/en-us/library/windows/desktop/ff191014(v=vs.85).aspx/.

[9]

M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP'03.

Digital Library

[10]

G. Ammons, T. Ball, and J. R. Larus. Exploiting hardware performance counters with flow and context sensitive profiling. In PLDI'97.

Digital Library

[11]

P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using magpie for request extraction and workload modelling. In OSDI'04.

Digital Library

[12]

M. D. Bond, G. Z. Baker, and S. Z. Guyer. Breadcrumbs: efficient context sensitivity for dynamic bug detection analyses. In PLDI'10.

Digital Library

[13]

M. D. Bond and K. S. McKinley. Probabilistic calling context. In OOPSLA '07.

Digital Library

[14]

M. Buchanan and A. A. Chien. Coordinated thread scheduling for workstation clusters under windows nt. In Proceedings of the USENIX Windows NT Workshop 1997.

Digital Library

[15]

B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In USENIX'04.

Digital Library

[16]

M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer. Pinpoint: Problem determination in large, dynamic internet services. In DSN'02.

Digital Library

[17]

T. Chilimbi, B. Liblit, K. Mehra, A. Nori, and K. Vaswani. Holmes: Effective statistical debugging via efficient path profiling. In ICSE'09.

Digital Library

[18]

U. Erlingsson, M. Peinado, S. Peter, and M. Budiu. Fay: extensible distributed tracing from kernels to clusters. In SOSP '11.

Digital Library

[19]

S. L. Graham, P. B. Kessler, and M. K. McKusick. gprof: a call graph execution profiler. SIGPLAN Not., 39(4):49--57, Apr. 2004.

Digital Library

[20]

S. Han, Y. Dang, S. Ge, D. Zhang, and T. Xie. Performance debugging in the large via mining millions of stack traces. In ICSE'12.

Digital Library

[21]

G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and detecting real-world performance bugs. In PLDI'12.

Digital Library

[22]

B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In PLDI'05.

Digital Library

[23]

C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In PLDI '05.

Digital Library

[24]

M. L. Massie, B. N. Chun, and D. E. Culler. The ganglia distributed monitoring system: Design, implementation and experience. In Parallel Computing, 2004.

[25]

R. J. Moore. A universal dynamic trace for linux and other operating systems. In Proceedings of the FREENIX Track: 2001 USENIX Annual Technical Conference, 2001.

Digital Library

[26]

N. Nethercote and J. Seward. Valgrind: a framework for heavyweight dynamic binary instrumentation. In PLDI '07.

Digital Library

[27]

V. Prasad, W. Cohen, F. C. Eigler, M. Hunt, J. Keniston, and B. Chen. Locating system problems using dynamic instrumentation. In Proceedings of the 2005 Ottawa Linux Symposium (OLS), 2005.

[28]

P. Reynolds, C. Killian, J. L. Wiener, J. C. Mogul, M. A. Shah, and A. Vahdat. Pip: detecting the unexpected in distributed systems. In NSDI'06.

Digital Library

[29]

B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag. Dapper, a large-scale distributed systems tracing infrastructure. Technical report, Google, Inc., 2010.

[30]

W. Sumner, Y. Zheng, D. Weeratunge, and X. Zhang. Precise calling context encoding. Software Engineering, IEEE Transactions on, 38(5), 2012.

Digital Library

[31]

B. C. Tak, C. Tang, C. Zhang, S. Govindan, B. Urgaonkar, and R. N. Chang. vpath: precise discovery of request processing paths from black-box observations of thread and network activities. In USENIX '09.

Digital Library

[32]

X. Xiao, S. Han, D. Zhang, and T. Xie. Context-sensitive delta inference for identifying workload-dependent performance bottlenecks. In ISSTA'13.

Digital Library

[33]

D. Yuan, H. Mai, W. Xiong, L. Tan, Y. Zhou, and S. Pasupathy. Sherlog: Error diagnosis by connecting clues from run-time logs. In ASPLOS'10.

Digital Library

[34]

D. Yuan, J. Zheng, S. Park, Y. Zhou, and S. Savage. Improving software diagnosability via log enhancement. In ASPLOS'11.

Digital Library

[35]

X. Zhuang, M. J. Serrano, H. W. Cain, and J.-D. Choi. Accurate, efficient, and adaptive calling context profiling. In PLDI'06.

Digital Library

Cited By

Shen JZhang HXiang YShi XLi XShen YZhang ZWu YYin XWang JXu MLi YYin JSong JLi ZNie RSchulzrinne HKohler EMaltz DMisra V(2023)Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero CodeProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604823(420-437)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604823
Gottschall BEeckhout LJahre MSolihin YHeinrich M(2023)TEA: Time-Proportional Event AnalysisProceedings of the 50th Annual International Symposium on Computer Architecture10.1145/3579371.3589058(1-13)Online publication date: 17-Jun-2023
https://dl.acm.org/doi/10.1145/3579371.3589058
Qin BTu TLiu ZYu TSong L(2022)Algorithmic Profiling for Real-World Complexity ProblemsIEEE Transactions on Software Engineering10.1109/TSE.2021.306765248:7(2680-2694)Online publication date: 1-Jul-2022
https://doi.org/10.1109/TSE.2021.3067652
Show More Cited By

Index Terms

IntroPerf: transparent context-sensitive multi-layer performance inference using system stack traces
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
    2. Metrics

Recommendations

IntroPerf: transparent context-sensitive multi-layer performance inference using system stack traces
Performance evaluation review

Performance bugs are frequently observed in commodity software. While profilers or source code-based tools can be used at development stage where a program is diagnosed in a well-defined environment, many performance bugs survive such a stage and affect ...
Performance Inference: A Novel Approach for Planning the Capacity of IaaS Cloud Applications
CLOUD '15: Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing

This work presents a novel approach to support application capacity planning in infrastructure-as-a-service (IaaS) clouds. The approach, called performance inference, relies on the assumption that it is possible to establish a capacity relation between ...
Nosv

nOSV can provide a bare-metal like performance for HPC applications on Cloud.The CPU cores and main memory are not shared among guest VMs of nOSV.Dedicated I/O resources are allocated to I/O sensitive HPC guests.Other virtualization environments can run ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMETRICS '14: The 2014 ACM international conference on Measurement and modeling of computer systems

June 2014

614 pages

ISBN:9781450327893

DOI:10.1145/2591971

General Chairs:
Sujay Sanghavi
The University of Texas at Austin
,
Sanjay Shakkottai
The University of Texas at Austin
,
Program Chairs:
Marc Lelarge
INRIA, France
,
Bianca Schroeder
University of Toronto

ACM SIGMETRICS Performance Evaluation Review Volume 42, Issue 1
Performance evaluation review
June 2014
569 pages
ISSN:0163-5999
DOI:10.1145/2637364
Editors:
Derek Eager
University of Saskatchewan
,
Carey Williamson
University of Calgary
Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMETRICS '14

Sponsor:

SIGMETRICS

SIGMETRICS '14: ACM SIGMETRICS / International Conference on Measurement and Modeling of Computer Systems

June 16 - 20, 2014

Texas, Austin, USA

Acceptance Rates

SIGMETRICS '14 Paper Acceptance Rate 40 of 237 submissions, 17%;

Overall Acceptance Rate 459 of 2,691 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
354
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shen JZhang HXiang YShi XLi XShen YZhang ZWu YYin XWang JXu MLi YYin JSong JLi ZNie RSchulzrinne HKohler EMaltz DMisra V(2023)Network-Centric Distributed Tracing with DeepFlow: Troubleshooting Your Microservices in Zero CodeProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604823(420-437)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604823
Gottschall BEeckhout LJahre MSolihin YHeinrich M(2023)TEA: Time-Proportional Event AnalysisProceedings of the 50th Annual International Symposium on Computer Architecture10.1145/3579371.3589058(1-13)Online publication date: 17-Jun-2023
https://dl.acm.org/doi/10.1145/3579371.3589058
Qin BTu TLiu ZYu TSong L(2022)Algorithmic Profiling for Real-World Complexity ProblemsIEEE Transactions on Software Engineering10.1109/TSE.2021.306765248:7(2680-2694)Online publication date: 1-Jul-2022
https://doi.org/10.1109/TSE.2021.3067652
Gottschall BEeckhout LJahre M(2021)TIP: Time-Proportional Instruction ProfilingMICRO-54: 54th Annual IEEE/ACM International Symposium on Microarchitecture10.1145/3466752.3480058(15-27)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3466752.3480058
Hong DNikravesh AMao ZKetkar MKishinevsky MTilevich E(2019)PerfProbeProceedings of the 6th International Conference on Mobile Software Engineering and Systems10.5555/3340730.3340743(50-61)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.5555/3340730.3340743
Hong DNikravesh AMao ZKetkar MKishinevsky M(2019)PerfProbe: A Systematic, Cross-Layer Performance Diagnosis Framework for Mobile Platforms2019 IEEE/ACM 6th International Conference on Mobile Software Engineering and Systems (MOBILESoft)10.1109/MOBILESoft.2019.00018(50-61)Online publication date: May-2019
https://doi.org/10.1109/MOBILESoft.2019.00018
Song LLu SUchitel SOrso ARobillard M(2017)Performance diagnosis for inefficient loopsProceedings of the 39th International Conference on Software Engineering10.1109/ICSE.2017.41(370-380)Online publication date: 20-May-2017
https://dl.acm.org/doi/10.1109/ICSE.2017.41
Pei KGu ZSaltaformaggio BMa SWang FZhang ZSi LZhang XXu DSchwab SRobertson WBalzarotti D(2016)HERCULEProceedings of the 32nd Annual Conference on Computer Security Applications10.1145/2991079.2991122(583-595)Online publication date: 5-Dec-2016
https://dl.acm.org/doi/10.1145/2991079.2991122
Kim CRhee JLee KZhang XXu DZimmermann TCleland-Huang JSu Z(2016)PerfGuard: binary-centric application performance monitoring in production environmentsProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering10.1145/2950290.2950347(595-606)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1145/2950290.2950347
Xu HNing XZhang HRhee JJiang G(2016)PInfer: Learning to Infer Concurrent Request Paths from System Kernel Events2016 IEEE International Conference on Autonomic Computing (ICAC)10.1109/ICAC.2016.38(199-208)Online publication date: Jul-2016
https://doi.org/10.1109/ICAC.2016.38
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents