research-article

Public Access

SourcererCC and SourcererCC-I: tools to detect clones in batch mode and during software development

Authors:

Hitesh Sajnani,

Cristina LopesAuthors Info & Claims

ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion

Pages 597 - 600

https://doi.org/10.1145/2889160.2889165

Published: 14 May 2016 Publication History

Abstract

Given the availability of large source-code repositories, there has been a large number of applications for large-scale clone detection. Unfortunately, despite a decade of active research, there is a marked lack in clone detectors that scale to big software systems or large repositories, specifically for detecting near-miss (Type 3) clones where significant editing activities may take place in the cloned code.

This paper demonstrates: (i) SourcererCC, a token-based clone detector that targets the first three clone types, and exploits an index to achieve scalability to large inter-project repositories using a standard workstation. It uses an optimized inverted-index to quickly query the potential clones of a given code block. Filtering heuristics based on token ordering are used to significantly reduce the size of the index, the number of code-block comparisons needed to detect the clones, as well as the number of required token-comparisons needed to judge a potential clone; and (ii) SourcererCC-I, an Eclipse plug-in, that uses SourcererCC's core engine to identify and navigate clones (both inter and intra project) in real-time during software development.

In our experiments, comparing SourcererCC with the state-of-the-art tools ¹, we found that it is the only clone detection tool to successfully scale to 250 MLOC on a standard workstation with 12 GB RAM and efficiently detect the first three types of clones (precision 86% and recall 86-100%). Link to the demo: https://youtu.be/17F_9Qp-ks4

References

[1]

C. K. Roy and J. R. Cordy, "A survey on software clone detection research," no. TR 2007-541, 2007. 115 pp.

[2]

T. Ishihara, K. Hotta, Y. Higo, H. Igaki, and S. Kusumoto, "Inter-project functional clone detection toward building libraries - an empirical study on 13,000 projects," in Reverse Engineering (WCRE), 2012 19th Working Conference on, pp. 387--391, Oct 2012.

Digital Library

[3]

K. Chen, P. Liu, and Y. Zhang, "Achieving accuracy and scalability simultaneously in detecting application clones on android markets," in Proceedings of the 36th International Conference on Software Engineering, ICSE 2014, (New York, NY, USA), pp. 175--186, ACM, 2014.

Digital Library

[4]

R. Koschke, "Large-scale inter-system clone detection using suffix trees," in Proceedings of CSMR, pp. 309--318, 2012.

Digital Library

[5]

A. Hemel and R. Koschke, "Reverse engineering variability in source code using clone detection: A case study for linux variants of consumer electronic devices," in Proceedings of Working Conference on Reverse Engineering, pp. 357--366, 2012.

Digital Library

[6]

J. Davies, D. German, M. Godfrey, and A. Hindle, "Software Bertillonage: finding the provenance of an entity," in Proceedings of MSR, 2011.

Digital Library

[7]

I. Keivanloo, J. Rilling, and P. Charland, "Internet-scale real-time code clone search via multi-level indexing," in Proceedings of WCRE, 2011.

Digital Library

[8]

S. Kawaguchi, T. Yamashina, H. Uwano, K. Fushida, Y. Kamei, M. Nagura, and H. Iida, "Shinobi: A tool for automatic code clone detection in the ide," vol. 0, (Los Alamitos, CA, USA), pp. 313--314, IEEE Computer Society, 2009.

Digital Library

[9]

J. Svajlenko, I. Keivanloo, and C. Roy, "Scaling classical clone detection tools for ultra-large datasets: An exploratory study," in Software Clones (IWSC), 2013 7th International Workshop on, pp. 16--22, May 2013.

Digital Library

[10]

S. Livieri, Y. Higo, M. Matsushita, and K. Inoue, "Very-large scale code clone analysis and visualization of open source programs using distributed ccfinder: D-ccfinder," in Proceedings of ICSE, 2007.

Digital Library

[11]

B. Hummel, E. Juergens, L. Heinemann, and M. Conradt, "Index-based code clone detection:incremental, distributed, scalable," in Proceedings of ICSM, 2010.

Digital Library

[12]

C. K. Roy and J. R. Cordy, "Near-miss function clones in open source software: An empirical study," J. Softw. Maint. Evol., vol. 22, pp. 165--189, Apr. 2010.

Digital Library

[13]

H. Sajnani, V. Saini, and C. Lopes, "A parallel and efficient approach to large scale clone detection," Journal of Software: Evolution and Process, vol. 27, no. 6, pp. 402--429, 2015.

Digital Library

[14]

H. Sajnani, V. Saini, J. Svajlenko, C. K. Roy, and C. V. Lopes, "Sourcerercc:scaling code clone detection to big-code," in Proceedings of the 38th International Conference on Software Engineering, 2016.

Digital Library

[15]

C. K. Roy, J. R. Cordy, and R. Koschke, "Comparison and evaluation of code clone detection techniques and tools: A qualitative approach," Sci. of Comput. Program., pp. 577--591, 2009.

Digital Library

[16]

B. Lague, D. Proulx, J. Mayrand, E. M. Merlo, and J. Hudepohl, "Assessing the benefits of incorporating function clone detection in a development process," in Software Maintenance, 1997. Proceedings., International Conference on, pp. 314--321, IEEE, 1997.

Digital Library

[17]

D. Hou, P. Jablonski, and F. Jacob, "Cnp: Towards an environment for the proactive management of copy-and-paste programming," in Program Comprehension, 2009. ICPC'09. IEEE 17th International Conference on, pp. 238--242, IEEE, 2009.

[18]

Z. Li, S. Lu, S. Myagmar, and Y. Zhou, "Cp-miner: A tool for finding copy-paste and related bugs in operating system code.," in OSDI, vol. 4, pp. 289--302, 2004.

Digital Library

[19]

E. Duala-Ekoko and M. P. Robillard, "Clonetracker: tool support for code clone management," in Proceedings of the 30th international conference on Software engineering, pp. 843--846, ACM, 2008.

Digital Library

[20]

B. S. Baker, "On finding duplication and near-duplication in large software systems," in Reverse Engineering, 1995., Proceedings of 2nd Working Conference on, pp. 86--95, IEEE, 1995.

Digital Library

[21]

N. Gode and R. Koschke, "Incremental clone detection," in Software Maintenance and Reengineering, 2009. CSMR'09. 13th European Conference on, pp. 219--228, IEEE, 2009.

Digital Library

[22]

T. Kamiya, S. Kusumoto, and K. Inoue, "CCFinder: a multilinguistic token-based code clone detection system for large scale source code," IEEE Trans. Softw. Eng., vol. 28, no. 7, pp. 654--670, 2002.

Digital Library

[23]

C. K. Roy and J. R. Cordy, "Nicad: Accurate detection of near-miss intentional clones using flexible pretty-printing and code normalization," in Program Comprehension, 2008. ICPC 2008. The 16th IEEE International Conference on, pp. 172--181, IEEE, 2008.

Digital Library

[24]

I. D. Baxter, A. Yahin, L. Moura, M. Sant'Anna, and L. Bier, "Clone detection using abstract syntax trees," in Software Maintenance, 1998. Proceedings., International Conference on, pp. 368--377, IEEE, 1998.

Digital Library

Cited By

Quradaa FShahzad SSaeed RSufyan M(2024)A novel code representation for detecting Java code clones using high-level and abstract compiled code representationsPLOS ONE10.1371/journal.pone.030233319:5(e0302333)Online publication date: 10-May-2024
https://doi.org/10.1371/journal.pone.0302333
Swilam ZHamdy APester A(2023)Cross-Language Code Clone Detection Using Abstract Syntax Tree and Graph Neural Network2023 International Conference on Computer and Applications (ICCA)10.1109/ICCA59364.2023.10401783(1-5)Online publication date: 28-Nov-2023
https://doi.org/10.1109/ICCA59364.2023.10401783
Zakeri-Nasrabadi MParsa SRamezani MRoy CEkhtiarzadeh M(2023)A systematic literature review on source code similarity measurement and clone detectionJournal of Systems and Software10.1016/j.jss.2023.111796204:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111796
Show More Cited By

Recommendations

SourcererCC: scaling code clone detection to big-code
ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Despite a decade of active research, there has been a marked lack in clone detection techniques that scale to large repositories for detecting near-miss clones. In this paper, we present a token-based clone detector, SourcererCC, that can detect both ...
CCAligner: a token based large-gap clone detector
ICSE '18: Proceedings of the 40th International Conference on Software Engineering

Copying code and then pasting with large number of edits is a common activity in software development, and the pasted code is a kind of complicated Type-3 clone. Due to large number of edits, we consider the clone as a large-gap clone. Large-gap clone ...
An extended assessment of type-3 clones as detected by state-of-the-art tools

Code reuse through copying and pasting leads to so-called software clones. These clones can be roughly categorized into identical fragments (type-1 clones), fragments with parameter substitution (type-2 clones), and similar fragments that differ through ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '16: Proceedings of the 38th International Conference on Software Engineering Companion

May 2016

946 pages

ISBN:9781450342056

DOI:10.1145/2889160

General Chair:
Laura Dillon
Michigan State University
,
Program Chairs:
Willem Visser
Stellenbosch University, South Africa
,
Laurie Williams
North Carolina State University

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\TCSE: TC on Software Engineering
IEEE-CS\DATC: IEEE Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\TCSE
IEEE-CS\DATC

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
603
Total Downloads

Downloads (Last 12 months)108
Downloads (Last 6 weeks)18

Reflects downloads up to 06 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Quradaa FShahzad SSaeed RSufyan M(2024)A novel code representation for detecting Java code clones using high-level and abstract compiled code representationsPLOS ONE10.1371/journal.pone.030233319:5(e0302333)Online publication date: 10-May-2024
https://doi.org/10.1371/journal.pone.0302333
Swilam ZHamdy APester A(2023)Cross-Language Code Clone Detection Using Abstract Syntax Tree and Graph Neural Network2023 International Conference on Computer and Applications (ICCA)10.1109/ICCA59364.2023.10401783(1-5)Online publication date: 28-Nov-2023
https://doi.org/10.1109/ICCA59364.2023.10401783
Zakeri-Nasrabadi MParsa SRamezani MRoy CEkhtiarzadeh M(2023)A systematic literature review on source code similarity measurement and clone detectionJournal of Systems and Software10.1016/j.jss.2023.111796204:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111796
Fang DLiu SLiu A(2022)Gated Homogeneous Fusion Networks With Jointed Feature Extraction for Defect PredictionIEEE Transactions on Reliability10.1109/TR.2022.316511571:2(512-526)Online publication date: Jun-2022
https://doi.org/10.1109/TR.2022.3165115
Patel SSinha R(2022)Combining Holistic Source Code Representation with Siamese Neural Networks for Detecting Code ClonesTesting Software and Systems10.1007/978-3-031-04673-5_12(148-159)Online publication date: 10-May-2022
https://doi.org/10.1007/978-3-031-04673-5_12
Wang HZhuang WZhang X(2021)Software Defect Prediction Based on Gated Hierarchical LSTMsIEEE Transactions on Reliability10.1109/TR.2020.304739670:2(711-727)Online publication date: Jun-2021
https://doi.org/10.1109/TR.2020.3047396
Sheneamer ARoy SKalita J(2021)An Effective Semantic Code Clone Detection Framework Using Pairwise Feature FusionIEEE Access10.1109/ACCESS.2021.30791569(84828-84844)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3079156
Walker ACerny TSong E(2020)Open-source tools and benchmarks for code-clone detectionACM SIGAPP Applied Computing Review10.1145/3381307.338131019:4(28-39)Online publication date: 28-Jan-2020
https://dl.acm.org/doi/10.1145/3381307.3381310
Sheneamer A(2020)An Automatic Advisor for Refactoring Software Clones Based on Machine LearningIEEE Access10.1109/ACCESS.2020.30061788(124978-124988)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3006178
Bharti SSingh H(2020)Proactively managing clones inside an IDE: a systematic literature reviewInternational Journal of Computers and Applications10.1080/1206212X.2020.172095244:3(230-249)Online publication date: 3-Feb-2020
https://doi.org/10.1080/1206212X.2020.1720952
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten