research-article

Efficient data race detection for distributed memory parallel programs

Authors:

Chang-Seo Park,

Costin IancuAuthors Info & Claims

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 51, Pages 1 - 12

https://doi.org/10.1145/2063384.2063452

Published: 12 November 2011 Publication History

Abstract

In this paper we present a precise data race detection technique for distributed memory parallel programs. Our technique, which we call Active Testing, builds on our previous work on race detection for shared memory Java and C programs and it handles programs written using shared memory approaches as well as bulk communication. Active testing works in two phases: in the first phase, it performs an imprecise dynamic analysis of an execution of the program and finds potential data races that could happen if the program is executed with a different thread schedule. In the second phase, active testing re-executes the program by actively controlling the thread schedule so that the data races reported in the first phase can be confirmed. A key highlight of our technique is that it can scalably handle distributed programs with bulk communication and single- and splitphase barriers. Another key feature of our technique is that it is precise---a data race confirmed by active testing is an actual data race present in the program; however, being a testing approach, our technique can miss actual data races. We implement the framework for the UPC programming language and demonstrate scalability up to a thousand cores for programs with both fine-grained and bulk (MPI style) communication. The tool confirms previously known bugs and uncovers several unknown ones. Our extensions capture constructs proposed in several modern programming languages for High Performance Computing, most notably non-blocking barriers and collectives.

References

[1]

S. V. Adve, M. D. Hill, B. P. Miller, and R. H. B. Netzer. Detecting data races on weak memory systems. In 18th International Symposium on Computer architecture (ISCA), pages 234--243. ACM, 1991.

Digital Library

[2]

R. Agarwal, A. Sasturkar, L. Wang, and S. D. Stoller. Optimized run-time race detection and atomicity checking using partial discovered types. In 20th International Conference on Automated software engineering (ASE), pages 233--242. ACM, 2005.

Digital Library

[3]

A. Aiken and D. Gay. Barrier inference. In Principles of programming languages (POPL), pages 342--354, New York, NY, USA, 1998. ACM.

Digital Library

[4]

S. Bensalem, J.-C. Fernandez, K. Havelund, and L. Mounier. Confirmation of deadlock potentials detected by runtime analysis. In PADTAD'06, pages 41--50, 2006.

Digital Library

[5]

S. Bensalem and K. Havelund. Scalable dynamic deadlock analysis of multi-threaded programs. In Parallel and Distributed Systems: Testing and Debugging 2005 (PADTAD'05), 2005.

[6]

Berkeley UPC. Available at http://upc.lbl.gov/.

[7]

S. Burckhardt, R. Alur, and M. M. K. Martin. CheckFence: checking consistency of concurrent data types on relaxed memory models. In Programming Language Design and Implementation (PLDI), pages 12--21, 2007.

Digital Library

[8]

J. Burnim, C. Stergiou, and K. Sen. Testing concurrent programs on relaxed memory models. In International Symposium on Software Testing and Analysis, 2011.

Digital Library

[9]

W. W. Carlson, J. M. Draper, D. E. Culler, K. Yelick, and K. W. E. Brooks. Introduction to UPC and language specification, 1999.

[10]

J.-D. Choi, K. Lee, A. Loginov, R. O'Callahan, V. Sarkar, and M. Sridharan. Efficient and precise datarace detection for multithreaded object-oriented programs. In Programming language design and implementation (PLDI), pages 258--269, New York, NY, USA, 2002. ACM.

Digital Library

[11]

J.-D. Choi, B. P. Miller, and R. H. B. Netzer. Techniques for debugging parallel programs with flowback analysis. ACM Trans. Program. Lang. Syst., 13(4):491--530, 1991.

Digital Library

[12]

M. Christiaens and K. D. Bosschere. TRaDe, a topological approach to on-the-fly race detection in java programs. In JavaTM Virtual Machine Research and Technology Symposium, pages 15--15. USENIX, 2001.

Digital Library

[13]

T. Cormen, C. Leiserson, and R. Rivset. Introduction to Algorithms. The MIT Press, 1994.

Digital Library

[14]

J. DeSouza, B. Kuhn, B. R. de Supinski, V. Samofalov, S. Zheltov, and S. Bratanov. Automated, scalable debugging of MPI programs with Intel Message Checker. In Software engineering for high performance computing system applications, SE-HPCS '05, pages 78--82, New York, NY, USA, 2005. ACM.

Digital Library

[15]

J. Dinan, P. Balaji, E. Lusk, P. Sadayappan, and R. Thakur. Hybrid parallel programming with MPI and Unified Parallel C. In Computing frontiers (CF), CF '10, 2010.

Digital Library

[16]

A. Dinning and E. Schonberg. Detecting access anomalies in programs with critical sections. In Workshop on Parallel and Distributed Debugging, 1991.

Digital Library

[17]

M. B. Dwyer, J. Hatcliff, Robby, and V. P. Ranganath. Exploiting object escape and locking information in partial-order reductions for concurrent object-oriented programs. Form. Methods Syst. Des., 25(2--3):199--240, 2004.

Digital Library

[18]

J. E. M. Clarke, O. Grumberg, and D. A. Peled. Model checking. MIT Press, 1999.

Digital Library

[19]

A. Farzan and P. Madhusudan. Monitoring atomicity in concurrent programs. In Computer Aided Verification (CAV), pages 52--65. Springer, 2008.

Digital Library

[20]

A. Farzan and P. Madhusudan. The complexity of predicting atomicity violations. In Tools and Algorithms for the Construction and Analysis of Systems (TACAS), 2009.

Digital Library

[21]

A. Farzan, P. Madhusudan, and F. Sorrentino. Meta-analysis for atomicity violations under nested locking. In Computer Aided Verification (CAV), pages 248--262. Springer-Verlag, 2009.

Digital Library

[22]

C. Flanagan and S. N. Freund. Atomizer: a dynamic atomicity checker for multithreaded programs. In 31st Symposium on Principles of Programming Languages (POPL), pages 256--267, 2004.

Digital Library

[23]

C. Flanagan and S. N. Freund. FastTrack: efficient and precise dynamic race detection. In Programming language design and implementation (PLDI). ACM, 2009.

Digital Library

[24]

C. Flanagan, S. N. Freund, and J. Yi. Velodrome: a sound and complete dynamic atomicity checker for multithreaded programs. In Programming language design and implementation (PLDI), pages 293--303. ACM, 2008.

Digital Library

[25]

Franklin: Cray XT4. At http://www.nersc.gov/users/computational-systems/franklin/.

[26]

E. N. Hanson and T. Johnson. The interval skip list: A data structure for finding all intervals that overlap a point. In Workshop on Algorithms and Data Structures, pages 153--164. Springer, 1992.

[27]

K. Havelund. Using runtime analysis to guide model checking of Java programs. In 7th International SPIN Workshop on Model Checking and Software Verification, pages 245--264, 2000.

Digital Library

[28]

K. Havelund and T. Pressburger. Model Checking Java Programs using Java PathFinder. Int. Journal on Software Tools for Technology Transfer, 2(4):366--381, 2000.

[29]

T. A. Henzinger, R. Jhala, and R. Majumdar. Race checking by context inference. SIGPLAN Not., 39(6):1--13, 2004.

Digital Library

[30]

P. N. Hilfinger, D. Bonachea, D. Gay, S. Graham, B. Liblit, G. Pike, and K. Yelick. Titanium language reference manual. Technical report, Berkeley, CA, USA, 2001.

Digital Library

[31]

G. Holzmann. The Spin model checker. IEEE Transactions on Software Engineering, 23(5):279--295, 1997.

Digital Library

[32]

N. Jalbert and K. Sen. A trace simplification technique for effective debugging of concurrent programs. In Foundations of Software Engineering (FSE). ACM, 2010.

Digital Library

[33]

P. Joshi, M. Naik, C.-S. Park, and K. Sen. An extensible active testing framework for concurrent programs. In Computer Aided Verification (CAV), Lecture Notes in Computer Science. Springer, 2009.

Digital Library

[34]

P. Joshi, C.-S. Park, K. Sen, and M. Naik. A randomized dynamic program analysis technique for detecting real deadlocks. In Programming Language Design and Implementation (PLDI), 2009.

Digital Library

[35]

B. Krammer, M. Müller, and M. Resch. Runtime checking of MPI applications with MARMOT. In Mini-Symposium Tools Support for Parallel Programming, ParCo 2005, Malaga, Spain, September 12 - 16, 2005., 2005.

[36]

L. Lamport. Time, clocks, and the ordering of events in a distributed system. Commun. ACM, 21(7):558--565, 1978.

Digital Library

[37]

S. Lu, J. Tucek, F. Qin, and Y. Zhou. AVIO: detecting atomicity violations via access interleaving invariants. SIGARCH Comput. Archit. News, 34(5):37--48, 2006.

Digital Library

[38]

J. Mellor-Crummey. On-the-fly detection of data races for programs with nested fork-join parallelism. In Supercomputing, pages 24--33. ACM, 1991.

Digital Library

[39]

M. Musuvathi and S. Qadeer. Iterative context bounding for systematic testing of multithreaded programs. In Programming Language Design and Implementation (PLDI), 2007.

Digital Library

[40]

M. Musuvathi, S. Qadeer, T. Ball, G. Basler, P. A. Nainar, and I. Neamtiu. Finding and reproducing heisenbugs in concurrent programs. In Operating Systems Design and Implementation (OSDI), pages 267--280, 2008.

Digital Library

[41]

M. Naik, A. Aiken, and J. Whaley. Effective static race detection for Java. In ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 308--319, 2006.

Digital Library

[42]

S. Narayanasamy, Z. Wang, J. Tigani, A. Edwards, and B. Calder. Automatically classifying benign and harmful data races using replay analysis. In Programming Language Design and Implementation (PLDI), pages 22--31, 2007.

Digital Library

[43]

The NAS Parallel Benchmarks. Available at http://www.nas.nasa.gov/Software/NPB.

[44]

The UPC NAS Parallel Benchmarks. Available at http://upc.gwu.edu/download.html.

[45]

Y. Nir-Buchbinder, R. Tzoref, and S. Ur. Deadlocks: From exhibiting to healing. In 8th Workshop on Runtime Verification, 2008.

Digital Library

[46]

H. Nishiyama. Detecting data races using dynamic escape analysis based on read barrier. In Virtual Machine Research and Technology Symposium, pages 127--138, 2004.

Digital Library

[47]

R. O'Callahan and J.-D. Choi. Hybrid dynamic data race detection. In Principles and practice of parallel programming (PPoPP), pages 167--178. ACM, 2003.

Digital Library

[48]

S. Olivier and J. Prins. Scalable dynamic load balancing using UPC. In International Conference on Parallel Processing (ICPP), ICPP '08, 2008.

Digital Library

[49]

C.-S. Park and K. Sen. Randomized active atomicity violation detection in concurrent programs. In Foundations of Software Engineering (FSE). ACM, 2008.

Digital Library

[50]

S. Park, S. Lu, and Y. Zhou. Ctrigger: exposing atomicity violation bugs from their hiding places. In Architectural support for programming languages and operating systems (ASPLOS), pages 25--36. ACM, 2009.

Digital Library

[51]

W. Pugh. Skip lists: a probabilistic alternative to balanced trees. Commun. ACM, 33(6):668--676, 1990.

Digital Library

[52]

S. Qadeer and D. Wu. KISS: keep it simple and sequential. In Programming language design and implementation (PLDI), pages 14--24. ACM, 2004.

Digital Library

[53]

M. Ronsse and K. D. Bosschere. Recplay: a fully integrated practical record/replay system. ACM Trans. Comput. Syst., 17(2):133--152, 1999.

Digital Library

[54]

S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: a dynamic data race detector for multithreaded programs. ACM Trans. Comput. Syst., 15(4):391--411, 1997.

Digital Library

[55]

E. Schonberg. On-the-fly detection of access anomalies. In Programming Language Design and Implementation (PLDI), volume 24, pages 285--297, 1989.

Digital Library

[56]

K. Sen. Race directed random testing of concurrent programs. In Programming Language Design and Implementation (PLDI), 2008.

Digital Library

[57]

O. Shacham, M. Sagiv, and A. Schuster. Scaling model checking of dataraces using dynamic information. J. Parallel Distrib. Comput., 67(5):536--550, 2007.

Digital Library

[58]

S. F. Siegel and T. K. Zirkel. Automatic formal verification of MPI-based parallel programs. In Principles and practice of parallel programming, PPoPP '11, pages 309--310, New York, NY, USA, 2011. ACM.

Digital Library

[59]

S. S. Vakkalanka, S. Sharma, G. Gopalakrishnan, and R. M. Kirby. ISP: a tool for model checking MPI programs. In Principles and practice of parallel programming, PPoPP '08, pages 285--286, New York, NY, USA, 2008. ACM.

Digital Library

[60]

J. S. Vetter and B. R. de Supinski. Dynamic software testing of MPI applications with Umpire. In Supercomputing, SC '00, Washington, DC, USA, 2000. IEEE Computer Society.

Digital Library

[61]

W. Visser, K. Havelund, G. Brat, and S. Park. Model checking programs. In Automated Software Engineering (ASE). IEEE, 2000.

Digital Library

[62]

A. Vo, S. Aananthakrishnan, G. Gopalakrishnan, B. R. d. Supinski, M. Schulz, and G. Bronevetsky. A scalable and distributed dynamic formal verifier for MPI programs. In Supercomputing, SC '10, pages 1--10, Washington, DC, USA, 2010. IEEE Computer Society.

Digital Library

[63]

C. von Praun and T. R. Gross. Object race detection. In Object oriented programming, systems, languages, and applications (OOPSLA), pages 70--82. ACM, 2001.

Digital Library

[64]

L. Wang and S. D. Stoller. Run-time analysis for atomicity. In 3rd Workshop on Run-time Verification, volume 89 of ENTCS, 2003.

[65]

L. Wang and S. D. Stoller. Accurate and efficient runtime detection of atomicity errors in concurrent programs. In Principles and Practice of Parallel Programming (PPoPP), pages 137--146. ACM Press, Mar. 2006.

Digital Library

[66]

A. Williams, W. Thies, and M. Ernst. Static deadlock detection for Java libraries. In European Conference on Object-Oriented Programming (ECOOP), pages 602--629, 2005.

Digital Library

[67]

M. Xu, R. Bodík, and M. D. Hill. A serializability violation detector for shared-memory server programs. SIGPLAN Not., 40(6):1--14, 2005.

Digital Library

[68]

Y. Yang, X. Chen, G. Gopalakrishnan, and R. M. Kirby. Efficient stateful dynamic partial order reduction. In SPIN, Lecture Notes in Computer Science, pages 288--305. Springer, 2008.

Digital Library

[69]

J. Yi, C. Sadowski, and C. Flanagan. SideTrack: generalizing dynamic atomicity analysis. In 7th Workshop on Parallel and Distributed Systems, pages 1--10. ACM, 2009.

Digital Library

[70]

Y. Zhang and E. Duesterwald. Barrier matching for programs with textually unaligned barriers. In Principles and practice of parallel programming, PPoPP '07, 2007.

Digital Library

Cited By

Schwitanski SOraji YPätzold CJenke JTomski FMüller M(2024)RMASanitizer: Generalized Runtime Detection of Data Races in Remote Memory Access ApplicationsProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673109(833-844)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673109
Gerlits E(2024)RaceHunter Dynamic Data Race DetectorProgramming and Computer Software10.1134/S036176882470033650:6(467-481)Online publication date: 19-Nov-2024
https://doi.org/10.1134/S0361768824700336
Wang W(2023)MPIRace: A Static Data Race Detector for MPI ProgramsLanguages and Compilers for Parallel Computing10.1007/978-3-031-31445-2_6(73-90)Online publication date: 10-May-2023
https://doi.org/10.1007/978-3-031-31445-2_6
Show More Cited By

Index Terms

Efficient data race detection for distributed memory parallel programs

Recommendations

Parallel data race detection for task parallel programs with locks
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Programming with tasks is a promising approach to write performance portable parallel code. In this model, the programmer explicitly specifies tasks and the task parallel runtime employs work stealing to distribute tasks among threads. Similar to ...
Algorithms for Data-Race Detection in Multithreaded Programs
Data race avoidance and replay scheme for developing and debugging parallel programs on distributed shared memory systems

Distributed shared memory (DSM) allows parallel programs to run on distributed computers by simulating a global virtual shared memory, but data racing bugs may easily occur when the threads of a multi-threaded process concurrently access the physically ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '11: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis

November 2011

866 pages

ISBN:9781450307710

DOI:10.1145/2063384

Conference Chair:
Scott Lathrop
University of Chicago
,
Program Chairs:
Jim Costa
Sandia National Laboratories
,
William Kramer
National Center for Supercomputing Applications

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

SC '11

Sponsor:

SIGARCH
IEEE-CS

SC '11: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 18, 2011

Washington, Seattle

Acceptance Rates

SC '11 Paper Acceptance Rate 74 of 352 submissions, 21%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
302
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Schwitanski SOraji YPätzold CJenke JTomski FMüller M(2024)RMASanitizer: Generalized Runtime Detection of Data Races in Remote Memory Access ApplicationsProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673109(833-844)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673109
Gerlits E(2024)RaceHunter Dynamic Data Race DetectorProgramming and Computer Software10.1134/S036176882470033650:6(467-481)Online publication date: 19-Nov-2024
https://doi.org/10.1134/S0361768824700336
Wang W(2023)MPIRace: A Static Data Race Detector for MPI ProgramsLanguages and Compilers for Parallel Computing10.1007/978-3-031-31445-2_6(73-90)Online publication date: 10-May-2023
https://doi.org/10.1007/978-3-031-31445-2_6
Bai JChen QJiang ZLawall JHu S(2022)Hybrid Static-Dynamic Analysis of Data Races Caused by Inconsistent Locking Discipline in Device DriversIEEE Transactions on Software Engineering10.1109/TSE.2021.3138735(1-1)Online publication date: 2022
https://doi.org/10.1109/TSE.2021.3138735
Xu YZhou AAgrawal KLee I(2022)PINT: Parallel INTerval-Based Race Detector2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS53621.2022.00087(850-861)Online publication date: May-2022
https://doi.org/10.1109/IPDPS53621.2022.00087
Yuan MLee YZhang CLi YCai YZhao BCadar CZhang X(2021)RAProducer: efficiently diagnose and reproduce data race bugs for binaries via trace analysisProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3460319.3464831(593-606)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3460319.3464831
Kallas KNiksic FStanford CAlur R(2020)DiffStream: differential output testing for stream processing programsProceedings of the ACM on Programming Languages10.1145/34282214:OOPSLA(1-29)Online publication date: 13-Nov-2020
https://dl.acm.org/doi/10.1145/3428221
Abe THosking AFinocchi I(2019)A type system for data independence of loop iterations in a directive-based PGAS languageProceedings of the 16th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes10.1145/3357390.3361021(50-62)Online publication date: 21-Oct-2019
https://dl.acm.org/doi/10.1145/3357390.3361021
Chen QBai JJiang ZLawall JHu S(2019)Detecting Data Races Caused by Inconsistent Lock Protection in Device Drivers2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER.2019.8668017(366-376)Online publication date: Feb-2019
https://doi.org/10.1109/SANER.2019.8668017
Protze JSchulz MAhn DMüller MZhao MChandra ARamakrishnan L(2018)Thread-local concurrencyProceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing10.1145/3208040.3208056(144-155)Online publication date: 11-Jun-2018
https://dl.acm.org/doi/10.1145/3208040.3208056
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten