skip to main content
10.1145/3411764.3445048acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

NBSearch: Semantic Search and Visual Exploration of Computational Notebooks

Published: 07 May 2021 Publication History

Abstract

Code search is an important and frequent activity for developers using computational notebooks (e.g., Jupyter). The flexibility of notebooks brings challenges for effective code search, where classic search interfaces for traditional software code may be limited. In this paper, we propose, NBSearch, a novel system that supports semantic code search in notebook collections and interactive visual exploration of search results. NBSearch leverages advanced machine learning models to enable natural language search queries and intuitive visualizations to present complicated intra- and inter-notebook relationships in the returned results. We developed NBSearch through an iterative participatory design process with two experts from a large software company. We evaluated the models with a series of experiments and the whole system with a controlled user study. The results indicate the feasibility of our analytical pipeline and the effectiveness of NBSearch to support code search in large notebook collections.

Supplementary Material

VTT File (3411764.3445048_videofigurecaptions.vtt)
VTT File (3411764.3445048_videopreviewcaptions.vtt)
MP4 File (3411764.3445048_videofigure.mp4)
Supplemental video
MP4 File (3411764.3445048_videopreview.mp4)
Preview video

References

[1]
F. Balmas. 2004. Displaying dependence graphs: a hierarchical approach. Journal of Software Maintenance and Evolution: Research and Practice 16, 3(2004), 151–185. https://doi.org/10.1002/smr.291
[2]
Souti Chattopadhyay, Ishita Prasad, Austin Z. Henley, Anita Sarma, and Titus Barik. 2020. What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). 1–12. https://doi.org/10.1145/3313831.3376729
[3]
Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/1412.3555(2014). arxiv:1412.3555http://arxiv.org/abs/1412.3555
[4]
E. Clarkson, K. Desai, and J. Foley. 2009. ResultMaps: Visualization for Search Interfaces. IEEE Transactions on Visualization and Computer Graphics 15, 6(2009), 1057–1064. https://doi.org/10.1109/TVCG.2009.176
[5]
Google Cloud. Accessed in 2020. Evaluating models. https://cloud.google.com/translate/automl/docs/evaluate
[6]
UCSD Library Digital Collections. Accessed in 2020. Data from: Exploration and Explanation in Computational Notebooks. https://library.ucsd.edu/dc/object/bb2733859v
[7]
Cecilia di Sciascio, Vedran Sabol, and Eduardo E. Veas. 2016. Rank As You Go: User-Driven Exploration of Search Results. In Proceedings of the 21st International Conference on Intelligent User Interfaces(IUI ’16). 118–129. https://doi.org/10.1145/2856767.2856797
[8]
Marian Dörk, Nathalie Henry Riche, Gonzalo Ramos, and Susan Dumais. 2012. PivotPaths: Strolling through Faceted Information Spaces. IEEE Transactions on Visualization and Computer Graphics 18, 12(2012), 2709–2718. https://doi.org/10.1109/TVCG.2012.252
[9]
Mi Feng, Cheng Deng, Evan M. Peck, and Lane Harrison. 2018. The Effects of Adding Search Functionality to Interactive Visualizations on the Web. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3173574.3173711
[10]
E. Gomez-Nieto, F. S. Roman, P. Pagliosa, W. Casaca, E. S. Helou, M. C. F. de Oliveira, and L. G. Nonato. 2014. Similarity Preserving Snippet-Based Visualization of Web Search Results. IEEE Transactions on Visualization and Computer Graphics 20, 3(2014), 457–470. https://doi.org/10.1109/TVCG.2013.242
[11]
S. Gratzl, A. Lex, N. Gehlenborg, H. Pfister, and M. Streit. 2013. LineUp: Visual Analysis of Multi-Attribute Rankings. IEEE Transactions on Visualization and Computer Graphics 19, 12(2013), 2277–2286. https://doi.org/10.1109/TVCG.2013.173
[12]
Philip J. Guo and Margo Seltzer. 2012. BURRITO: Wrapping Your Lab Notebook in Computational Infrastructure. In Proceedings of the 4th USENIX Conference on Theory and Practice of Provenance(TaPP’12). 7.
[13]
Andrew Head, Fred Hohman, Titus Barik, Steven M. Drucker, and Robert DeLine. 2019. Managing Messes in Computational Notebooks. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. https://doi.org/10.1145/3290605.3300500
[14]
Andrew Head, Jason Jiang, James Smith, Marti A. Hearst, and Björn Hartmann. 2020. Composing Flexibly-Organized Step-by-Step Tutorials from Linked Source Code, Snippets, and Outputs. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3313831.3376798
[15]
Marti A. Hearst. 1995. TileBars: Visualization of Term Distribution Information in Full Text Information Access. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’95). 59–66. https://doi.org/10.1145/223904.223912
[16]
Desmond G. Higgins and Paul M. Sharp. 1988. CLUSTAL: a package for performing multiple sequence alignment on a microcomputer. Gene 73, 1 (1988), 237 – 244. https://doi.org/10.1016/0378-1119(88)90330-7
[17]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (1997), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
[18]
Jane Hoffswell, Arvind Satyanarayan, and Jeffrey Heer. 2018. Augmenting Code with In Situ Visualizations to Aid Program Understanding. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. https://doi.org/10.1145/3173574.3174106
[19]
Hugging Face. Accessed in 2020. Pretrained Models. https://huggingface.co/transformers/pretrained_models.html
[20]
Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arxiv:1502.03167 [cs.LG]
[21]
Mary Beth Kery, Amber Horvath, and Brad Myers. 2017. Variolite: Supporting Exploratory Programming by Data Scientists. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(CHI ’17). 1265–1276. https://doi.org/10.1145/3025453.3025626
[22]
Mary Beth Kery, Bonnie E. John, Patrick O’Flaherty, Amber Horvath, and Brad A. Myers. 2019. Towards Effective Foraging by Data Scientists to Find Past Analysis Choices. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems(CHI ’19). 1–13. https://doi.org/10.1145/3290605.3300322
[23]
Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E. John, and Brad A. Myers. 2018. The Story in the Notebook: Exploratory Data Science using a Literate Programming Tool. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. https://doi.org/10.1145/3173574.3173748
[24]
Yanir Kleiman, Joel Lanir, Dov Danon, Yasmin Felberbaum, and Daniel Cohen-Or. 2015. DynamicMaps: Similarity-based Browsing Through a Massive Set of Images. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems(CHI ’15). 995–1004. https://doi.org/10.1145/2702123.2702224
[25]
Ken Krugler. 2013. Krugle Code Search Architecture. Springer New York, 103–120. https://doi.org/10.1007/978-1-4614-6596-6_6
[26]
Byron Y-L Kuo, Thomas Hentrich, Benjamin M . Good, and Mark D. Wilkinson. 2007. Tag Clouds for Summarizing Web Search Results. In Proceedings of the 16th International Conference on World Wide Web(WWW ’07). 1203–1204. https://doi.org/10.1145/1242572.1242766
[27]
Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. arxiv:1405.4053 [cs.CL]
[28]
Otávio Augusto Lazzarini Lemos, Sushil Krishna Bajracharya, Joel Ossher, Ricardo Santos Morla, Paulo Cesar Masiero, Pierre Baldi, and Cristina Videira Lopes. 2007. CodeGenie. In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering - ASE '07. https://doi.org/10.1145/1321631.1321726
[29]
Erik Linstead, Sushil Bajracharya, Trung Ngo, Paul Rigor, Cristina Lopes, and Pierre Baldi. 2008. Sourcerer: mining and searching internet-scale software repositories. Data Mining and Knowledge Discovery 18, 2 (2008), 300–336. https://doi.org/10.1007/s10618-008-0118-x
[30]
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. CoRR abs/1508.04025(2015). arxiv:1508.04025http://arxiv.org/abs/1508.04025
[31]
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press.
[32]
Collin McMillan, Mark Grechanik, Denys Poshyvanyk, Chen Fu, and Qing Xie. 2012. Exemplar: A Source Code Search Engine for Finding Highly Relevant Applications. IEEE Transactions on Software Engineering 38, 5 (2012), 1069–1087. https://doi.org/10.1109/tse.2011.84
[33]
T. Nguyen and J. Zhang. 2006. A Novel Visualization Model for Web Search Results. IEEE Transactions on Visualization and Computer Graphics 12, 5(2006), 981–988. https://doi.org/10.1109/TVCG.2006.111
[34]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics(ACL ’02). 311–318. https://doi.org/10.3115/1073083.1073135
[35]
Jaakko Peltonen, Kseniia Belorustceva, and Tuukka Ruotsalo. 2017. Topic-Relevance Map: Visualization for Improving Search Result Comprehension. In Proceedings of the 22nd International Conference on Intelligent User Interfaces(IUI ’17). 611–622. https://doi.org/10.1145/3025171.3025223
[36]
Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora.
[37]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)(2019). https://doi.org/10.18653/v1/d19-1410
[38]
Adam Rule, Aurélien Tabard, and James D. Hollan. 2018. Exploration and Explanation in Computational Notebooks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. https://doi.org/10.1145/3173574.3173606
[39]
Caitlin Sadowski, Kathryn T. Stolee, and Sebastian Elbaum. 2015. How Developers Search for Code: A Case Study. Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (2015), 191–201. https://doi.org/10.1145/2786805.2786855
[40]
M. Schuster and K. K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing 45, 11 (1997), 2673–2681.
[41]
Doreen Seider, Andreas Schreiber, Tobias Marquardt, and Marlene Bruggemann. 2016. Visualizing Modules and Dependencies of OSGi-Based Applications. In Proceedings of the IEEE Working Conference on Software Visualization (VISSOFT). https://doi.org/10.1109/vissoft.2016.20
[42]
Alex Sherstinsky. 2018. Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network. CoRR abs/1808.03314(2018). arxiv:1808.03314http://arxiv.org/abs/1808.03314
[43]
Susan Elliott Sim, Medha Umarji, Sukanya Ratanotayanon, and Cristina V. Lopes. 2011. How Well Do Search Engines Support Code Retrieval on the Web?ACM Trans. Softw. Eng. Methodol. 21, 1, Article 4(2011), 25 pages. https://doi.org/10.1145/2063239.2063243
[44]
SourceForge. Accessed in 2020. . https://sourceforge.net/
[45]
A. Spoerri. 2004. RankSpiral: Toward Enhancing Search Results Visualizations. In Proceedings of IEEE Symposium on Information Visualization. 18–18. https://doi.org/10.1109/INFVIS.2004.56
[46]
Krishna Subramanian, Johannes Maas, and Jan Borchers. 2020. TRACTUS: Understanding and Supporting Source Code Experimentation in Hypothesis-Driven Data Science. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3313831.3376764
[47]
Jaime Teevan, Christine Alvarado, Mark S. Ackerman, and David R. Karger. 2004. The perfect search engine is not enough. In Proceedings of the 2004 conference on Human factors in computing systems - CHI '04. https://doi.org/10.1145/985692.985745
[48]
John Wenskovitch, Jian Zhao, Scott Carter, Matthew Cooper, and Chris North. 2019. Albireo: An Interactive Tool for Visually Summarizing Computational Notebook Structure. In Proceedings of the IEEE Symposium on Visualization in Data Science. 1–10. https://doi.org/10.1109/VDS48975.2019.8973385
[49]
Wikipedia contributors. Accessed in 2020. Koders — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Koders&oldid=869578010
[50]
Max L. Wilson, Bill Kules, m. c. schraefel, and Ben Shneiderman. 2010. From Keyword Search to Exploration: Designing Future Search Interfaces for the Web. Foundations and Trends® in Web Science 2, 1 (2010), 1–97. https://doi.org/10.1561/1800000003
[51]
Jian Zhao, Chidansh Bhatt, Matthew Cooper, and David A. Shamma. 2018. Flexible Learning with Semantic Visual Exploration and Sequence-Based Recommendation of MOOC Videos. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). 1–13. https://doi.org/10.1145/3173574.3173903
[52]
Jian Zhao, Christopher Collins, Fanny Chevalier, and Ravin Balakrishnan. 2013. Interactive Exploration of Implicit and Explicit Relations in Faceted Datasets. IEEE Transactions on Visualization and Computer Graphics 19, 12(2013), 2080–2089. https://doi.org/10.1109/TVCG.2013.167
[53]
Jian Zhao, Michael Glueck, Fanny Chevalier, Yanhong Wu, and Azam Khan. 2016. Egocentric Analysis of Dynamic Networks with EgoLines. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16. 5003–5014. https://doi.org/10.1145/2858036.2858488

Cited By

View all

Index Terms

  1. NBSearch: Semantic Search and Visual Exploration of Computational Notebooks
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
          May 2021
          10862 pages
          ISBN:9781450380966
          DOI:10.1145/3411764
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 07 May 2021

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. Semantic code search
          2. computational notebooks
          3. document and text analysis.
          4. search result visualization

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Funding Sources

          Conference

          CHI '21
          Sponsor:

          Acceptance Rates

          Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

          Upcoming Conference

          CHI 2025
          ACM CHI Conference on Human Factors in Computing Systems
          April 26 - May 1, 2025
          Yokohama , Japan

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)123
          • Downloads (Last 6 weeks)11
          Reflects downloads up to 26 Dec 2024

          Other Metrics

          Citations

          Cited By

          View all

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Media

          Figures

          Other

          Tables

          Share

          Share

          Share this Publication link

          Share on social media