Article

Free access

Semantics-assisted code review: an efficient toolchain and a user study

Authors:

Massimiliano Menarini,

William G. GriswoldAuthors Info & Claims

ASE '17: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering

Pages 554 - 565

Published: 30 October 2017 Publication History

Abstract

Code changes are often reviewed before they are deployed. Popular source control systems aid code review by presenting textual differences between old and new versions of the code, leaving developers with the difficult task of determining whether the differences actually produced the desired behavior. Fortunately, we can mine such information from code repositories. We propose aiding code review with inter-version semantic differential analysis. During review of a new commit, a developer is presented with summaries of both code differences and behavioral differences, which are expressed as diffs of likely invariants extracted by running the system's test cases. As a result, developers can more easily determine that the code changes produced the desired effect. We created an invariant-mining tool chain, GETTY, to support our concept of semantically-assisted code review. To validate our approach, 1) we applied GETTY to the commits of 6 popular open source projects, 2) we assessed the performance and cost of running GETTY in different configurations, and 3) we performed a comparative user study with 18 developers. Our results demonstrate that semantically-assisted code review is feasible, effective, and that real programmers can leverage it to improve the quality of their reviews.

References

[1]

O. Kononenko, O. Baysal, and M. W. Godfrey, “Code Review Quality: How Developers See It,” in Proceedings of the 38th International Conference on Software Engineering, New York, NY, USA, 2016, pp. 1028–1038.

Digital Library

[2]

“Git.” {Online}. Available: https://git-scm.com/.

[3]

“Git - git-diff Documentation.” {Online}. Available: https://gitscm.com/docs/git-diff.

[4]

G. Yang, S. Khurshid, S. Person, and N. Rungta, “Property Differencing for Incremental Checking,” in Proceedings of the 36th International Conference on Software Engineering, New York, NY, USA, 2014, pp. 1059–1070.

Digital Library

[5]

S. K. Lahiri, K. L. McMillan, R. Sharma, and C. Hawblitzel, “Differential Assertion Checking,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, New York, NY, USA, 2013, pp. 345–355.

Digital Library

[6]

M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, “Dynamically discovering likely program invariants to support program evolution,” IEEE Trans. Softw. Eng., vol. 27, no. 2, pp. 99–123, Feb. 2001.

Digital Library

[7]

“google/gson,” GitHub. {Online}. Available: https://github.com/google/gson. {Accessed: 11-May-2017}.

[8]

S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “The Impact of Code Review Coverage and Code Review Participation on Software Quality: A Case Study of the Qt, VTK, and ITK Projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories, New York, NY, USA, 2014, pp. 192–201.

Digital Library

[9]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-Directed Random Test Generation,” in Proceedings of the 29th International Conference on Software Engineering, Washington, DC, USA, 2007, pp. 75–84.

Digital Library

[10]

G. C. Murphy, W. G. Griswold, M. P. Robillard, J. Hannemann, and W. Leong, “Design recommendations for concern elaboration tools,” Asp.-Oriented Softw. Dev., pp. 507–530, 2005.

[11]

Y. Tao, Y. Dang, T. Xie, D. Zhang, and S. Kim, “How Do Software Engineers Understand Code Changes?: An Exploratory Study in Industry,” in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, New York, NY, USA, 2012, p. 51:1–51:11.

Digital Library

[12]

R. Holmes and D. Notkin, “Identifying program, test, and environmental changes that affect behaviour,” in 2011 33rd International Conference on Software Engineering (ICSE), 2011, pp. 371–380.

Digital Library

[13]

Y. Yan, M. Menarini, and W. Griswold, “Mining Software Contracts for Software Evolution,” in 2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 471–475.

Digital Library

[14]

“Apache Maven Project.” {Online}. Available: http://maven.apache.org/. {Accessed: 11-May-2017}.

[15]

“JUnit 4,” GitHub. {Online}. Available: https://github.com/junit-team/junit4. {Accessed: 11-May- 2017}.

[16]

P. Hirch, “Automatic inference of JML-based security specifications with exception handling,” Master Thesis, Universität Bremen, 2016.

[17]

“Apache Commons – Apache Commons.” {Online}. Available: https://commons.apache.org/. {Accessed: 11-May-2017}.

[18]

Y. Yan, “Getty Main Repository,” Getty - Semantiful Differentials. {Online}. Available: https://github.com/mmenarini/semantiful-differentials-getty. {Accessed: 06-Sep-2017}.

[19]

“Maven - Maven EMMA plugin.” {Online}. Available: http://emma.sourceforge.net/maven-emma-plugin/. {Accessed: 11-May-2017}.

[20]

L. Inozemtseva and R. Holmes, “Coverage is Not Strongly Correlated with Test Suite Effectiveness,” in Proceedings of the 36th International Conference on Software Engineering, New York, NY, USA, 2014, pp. 435–445.

Digital Library

[21]

Debug Team, “Eclipse Debug Project.” {Online}. Available: https://www.eclipse.org/eclipse/debug/. {Accessed: 11-May- 2017}.

[22]

“Git - git-blame Documentation.” {Online}. Available: https://git-scm.com/docs/git-blame. {Accessed: 11-May-2017}.

[23]

“{CLI-252} LongOpt falsely detected as ambiguous - ASF JIRA.” {Online}. Available: https://issues.apache.org/jira/browse/CLI-252. {Accessed: 11-May- 2017}.

[24]

H. J. Postel, “Die Kölner Phonetik. Ein Verfahren zur Identifizierung von Personennamen auf der Grundlage der Gestaltanalyse,” IBM-Nachrichten, vol. 19, pp. 925–931, 1969.

[25]

A. H. JØRGENSEN, “Thinking-aloud in user interface design: a method promoting cognitive ergonomics,” Ergonomics, vol. 33, no. 4, pp. 501–507, Apr. 1990.

[26]

N. Miyake, “Constructive Interaction and the Iterative Process of Understanding,” Cogn. Sci., vol. 10, no. 2, pp. 151–177, Apr. 1986.

[27]

“Added test to use BigDecimal to parse number when requesting it as a … · google/gson@e450822,” GitHub. {Online}. Available: https://github.com/google/gson/commit/e4508227c53749b48318366c1272119031851887. {Accessed: 11-May-2017}.

[28]

“LazilyParsedNumber does not implement eqals and hashCode methods · Issue #627 · google/gson,” GitHub. {Online}. Available: https://github.com/google/gson/issues/627. {Accessed: 11-May-2017}.

[29]

“Parse all JSON numbers as either BigDecimal or BigInteger. From ther… · google/gson@d5319d9,” GitHub. {Online}. Available: https://github.com/google/gson/commit/d5319d9e840b2c7237ca435f50c50ffbe7dce507. {Accessed: 11-May-2017}.

[30]

S. K. Lahiri, K. Vaswani, and C. A. R. Hoare, “Differential Static Analysis: Opportunities, Applications, and Challenges,” in Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research, New York, NY, USA, 2010, pp. 201–204.

Digital Library

[31]

S. Person, M. B. Dwyer, S. Elbaum, and C. S. Pǎsǎreanu, “Differential Symbolic Execution,” in Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, 2008, pp. 226–237.

Digital Library

[32]

J. Branchaud, S. Person, and N. Rungta, “A Change Impact Analysis to Characterize Evolving Program Behaviors,” in Proceedings of the 2012 IEEE International Conference on Software Maintenance (ICSM), Washington, DC, USA, 2012, pp. 109–118.

Digital Library

[33]

N. Rungta, S. Person, and J. Branchaud, “A Change Impact Analysis to Characterize Evolving Program Behaviors,” in 2012 28th IEEE International Conference on Software Maintenance (ICSM), 2012, pp. 109–118.

Digital Library

[34]

D. Qi, J. Yi, and A. Roychoudhury, “Software Change Contracts,” in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, New York, NY, USA, 2012, p. 22:1–22:4.

Digital Library

[35]

T. D. B. Le, J. Yi, D. Lo, F. Thung, and A. Roychoudhury, “Dynamic Inference of Change Contracts,” in 2014 IEEE International Conference on Software Maintenance and Evolution, 2014, pp. 451–455.

Digital Library

[36]

H. Palikareva, T. Kuchta, and C. Cadar, “Shadow of a Doubt: Testing for Divergences Between Software Versions,” in Proceedings of the 38th International Conference on Software Engineering, New York, NY, USA, 2016, pp. 1181–1192.

Digital Library

[37]

“Gerrit Code Review.” {Online}. Available: https://www.gerritcodereview.com/. {Accessed: 16-Nov-2016}.

Cited By

Badampudi DUnterkalmsteiner MBritto R(2023)Modern Code Reviews—Survey of Literature and PracticeACM Transactions on Software Engineering and Methodology10.1145/358500432:4(1-61)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3585004
Rowe RFérée HThompson SOwens SMcKinley KFisher K(2019)Characterising renaming within OCaml’s module system: theory and implementationProceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation10.1145/3314221.3314600(950-965)Online publication date: 8-Jun-2019
https://dl.acm.org/doi/10.1145/3314221.3314600

Recommendations

Understanding code snippets in code reviews: a preliminary study of the OpenStack community
ICPC '22: Proceedings of the 30th IEEE/ACM International Conference on Program Comprehension

Code review is a mature practice for software quality assurance in software development with which reviewers check the code that has been committed by developers, and verify the quality of code. During the code review discussions, reviewers and ...
When testing meets code review: why and how developers review tests
ICSE '18: Proceedings of the 40th International Conference on Software Engineering

Automated testing is considered an essential process for ensuring software quality. However, writing and maintaining high-quality test code is challenging and frequently considered of secondary importance. For production code, many open source and ...
Visualizing code and coverage changes for code review
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

One of the tasks of reviewers is to verify that code modifications are well tested. However, current tools offer little support in understanding precisely how changes to the code relate to changes to the tests. In particular, it is hard to see whether (...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ASE '17: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering

October 2017

1033 pages

ISBN:9781538626849

General Chair:
Grigore Rosu
University of Illinois at Urbana-Champaign, USA
,
Program Chairs:
Massimiliano Di Penta
University of Sannio, Italy
,
Tien N. Nguyen
University of Texas at Dallas, USA

Sponsors

Publisher

IEEE Press

Publication History

Published: 30 October 2017

Author Tags

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
220
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)6

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Badampudi DUnterkalmsteiner MBritto R(2023)Modern Code Reviews—Survey of Literature and PracticeACM Transactions on Software Engineering and Methodology10.1145/358500432:4(1-61)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3585004
Rowe RFérée HThompson SOwens SMcKinley KFisher K(2019)Characterising renaming within OCaml’s module system: theory and implementationProceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation10.1145/3314221.3314600(950-965)Online publication date: 8-Jun-2019
https://dl.acm.org/doi/10.1145/3314221.3314600

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents