research-article

OneLabeler: A Flexible System for Building Data Labeling Tools

Authors:

Dongmei ZhangAuthors Info & Claims

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Article No.: 93, Pages 1 - 22

https://doi.org/10.1145/3491102.3517612

Published: 28 April 2022 Publication History

Abstract

Labeled datasets are essential for supervised machine learning. Various data labeling tools have been built to collect labels in different usage scenarios. However, developing labeling tools is time-consuming, costly, and expertise-demanding on software development. In this paper, we propose a conceptual framework for data labeling and OneLabeler based on the conceptual framework to support easy building of labeling tools for diverse usage scenarios. The framework consists of common modules and states in labeling tools summarized through coding of existing tools. OneLabeler supports configuration and composition of common software modules through visual programming to build data labeling tools. A module can be a human, machine, or mixed computation procedure in data labeling. We demonstrate the expressiveness and utility of the system through ten example labeling tools built with OneLabeler. A user study with developers provides evidence that OneLabeler supports efficient building of diverse data labeling tools.

Supplementary Material

Supplemental Materials (3491102.3517612-supplemental-materials.zip)

Download
14.22 KB

MP4 File (3491102.3517612-video-preview.mp4)

Video Preview

Download
5.36 MB

MP4 File (3491102.3517612-talk-video.mp4)

Talk Video

Download
77.05 MB

References

[1]

Amazon. 2005. Amazon Mechanical Turk. https://www.mturk.com/

[2]

Mykhaylo Andriluka, Jasper R. R. Uijlings, and Vittorio Ferrari. 2018. Fluid Annotation: A Human-Machine Collaboration Interface for Full Image Annotation. In Proceedings of the ACM International Conference on Multimedia. ACM, 1957–1966.

Digital Library

[3]

Nathaniel Ayewah, William Pugh, David Hovemeyer, J. David Morgenthaler, and John Penix. 2008. Using Static Analysis to Find Bugs. IEEE Software 25, 5 (sep 2008), 22–29.

Digital Library

[4]

A. Bäuerle, H. Neumann, and T. Ropinski. 2020. Classifier-Guided Visual Correction of Noisy Labels for Image Classification Tasks. Computer Graphics Forum 39, 3 (jun 2020), 195–205.

[5]

Jürgen Bernard, Marco Hutter, Matthias Zeppelzauer, Dieter Fellner, and Michael Sedlmair. 2018. Comparing Visual-Interactive Labeling with Active Learning: An Experimental Study. IEEE Transactions on Visualization and Computer Graphics 24, 1 (Jan. 2018), 298–308.

[6]

Jürgen Bernard, Matthias Zeppelzauer, Michael Sedlmair, and Wolfgang Aigner. 2018. VIAL: a unified process for visual interactive labeling. The Visual Computer 34, 9 (2018), 1189–1207.

Digital Library

[7]

Michael Bloodgood and K. Vijay-Shanker. 2009. A Method for Stopping Active Learning Based on Stabilizing Predictions and the Need for User-Adjustable Stopping. In Proceedings of the Conference on Computational Natural Language Learning. Association for Computational Linguistics, 39–47.

[8]

Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D3: Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12 (Nov 2011), 2301–2309.

Digital Library

[9]

Yuri Boykov, Olga Veksler, and Ramin Zabih. 2001. Fast Approximate Energy Minimization via Graph Cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11(2001), 1222–1239.

Digital Library

[10]

Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie. 2010. Visual Recognition with Humans in the Loop. In Proceedings of the European Conference on Computer Vision. Springer Berlin Heidelberg, 438–451.

[11]

Klaus Brinker. 2003. Incorporating Diversity in Active Learning with Support Vector Machines. In Proceedings of the International Conference on Machine Learning. AAAI Press, 59–66.

[12]

Carla E. Brodley and Mark A. Friedl. 1999. Identifying Mislabeled Training Data. Journal of Artificial Intelligence Research 11, 1 (July 1999), 131–167.

[13]

Nicholas J. Bryan, Gautham J. Mysore, and Ge Wang. 2014. ISSE: An Interactive Source Separation Editor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 257–266.

Digital Library

[14]

Maya Cakmak and Andrea L. Thomaz. 2011. Mixed-Initiative Active Learning. In Proceedings of ICML Workshop on Combining Learning Strategies to Reduce Label Cost. 5 pages.

[15]

Chengliang Chai, Guoliang Li, Ju Fan, and Yuyu Luo. 2020. Crowdsourcing-based Data Extraction from Visualization Charts. In Proceedings of the IEEE International Conference on Data Engineering. IEEE, 1814–1817.

[16]

Joseph Chee Chang, Saleema Amershi, and Ece Kamar. 2017. Revolt: Collaborative Crowdsourcing for Labeling Machine Learning Datasets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2334–2346.

Digital Library

[17]

Justin Cheng and Michael S. Bernstein. 2015. Flock: Hybrid Crowd-Machine Learning Classifiers. In Proceedings of the ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 600–611.

Digital Library

[18]

Minsuk Choi, Cheonbok Park, Soyoung Yang, Yonggyu Kim, Jaegul Choo, and Sungsoo Ray Hong. 2019. AILA: Attentive Interactive Labeling Assistant for Document Classification through Attention-Based Deep Neural Networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1–12.

Digital Library

[19]

Jingyu Cui, Fang Wen, Rong Xiao, Yuandong Tian, and Xiaoou Tang. 2007. EasyAlbum: An Interactive Photo Annotation System Based on Face Clustering and Re-ranking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 367–376.

Digital Library

[20]

Weiwei Cui, Xiaoyu Zhang, Yun Wang, He Huang, Bei Chen, Lei Fang, Haidong Zhang, Jian-Guan Lou, and Dongmei Zhang. 2020. Text-to-Viz: Automatic Generation of Infographics from Proportion-Related Natural Language Statements. IEEE Transactions on Visualization and Computer Graphics 26, 1 (jan 2020), 906–916.

[21]

Ork de Rooij, Jarke J. van Wijk, and Marcel Worring. 2010. MediaTable: Interactive Categorization of Multimedia Collections. IEEE Computer Graphics and Applications 30, 5 (sep 2010), 42–51.

Digital Library

[22]

Jia Deng, Olga Russakovsky, Jonathan Krause, Michael Bernstein, Alex Berg, and Li Fei-Fei. 2014. Scalable Multi-Label Annotation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3099–3102.

Digital Library

[23]

Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml

[24]

Abhishek Dutta and Andrew Zisserman. 2019. The VIA Annotation Software for Images, Audio and Video. In Proceedings of the ACM International Conference on Multimedia. ACM, 2276–2279.

Digital Library

[25]

Sara Evensen, Chang Ge, and Cagatay Demiralp. 2020. Ruler: Data Programming by Demonstration for Document Labeling. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, 1996–2005.

[26]

Explosion. 2017. Prodigy. https://prodi.gy/

[27]

Jerry Alan Fails and Dan R. Olsen. 2003. Interactive Machine Learning. In Proceedings of the International Conference on Intelligent User Interfaces. ACM, 39–45.

Digital Library

[28]

Cristian Felix, Aritra Dasgupta, and Enrico Bertini. 2018. The Exploratory Labeling Assistant: Mixed-Initiative Label Curation with Large Document Collections. In Proceedings of the ACM Symposium on User Interface Software and Technology. ACM, 153–164.

Digital Library

[29]

Humberto S. Garcia Caballero, Michel A. Westenberg, Binyam Gebre, and Jarke J. van Wijk. 2019. V-Awake: A Visual Analytics Approach for Correcting Sleep Predictions from Deep Learning Models. Computer Graphics Forum 38, 3 (2019), 1–12.

[30]

Gaudenz Halter, Rafael Ballester-Ripoll, Barbara Flueckiger, and Renato Pajarola. 2019. VIAN: A Visual Annotation Tool for Film Analysis. Computer Graphics Forum 38, 3 (2019), 119–129.

[31]

Benjamin Höferlin, Rudolf Netzel, Markus Höferlin, Daniel Weiskopf, and Gunther Heidemann. 2012. Inter-Active Learning of Ad-Hoc Classifiers for Video Visual Analytics. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology. IEEE, 23–32.

Digital Library

[32]

Steven C.H. Hoi and Michael R. Lyu. 2005. A Semi-Supervised Active Learning Framework for Image Retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2. IEEE, 302–309.

[33]

Xian-Sheng Hua and Guo-Jun Qi. 2008. Online Multi-Label Active Annotation: Towards Large-Scale Content-Based Video Search. In Proceedings of the ACM International Conference on Multimedia. ACM, 141–150.

Digital Library

[34]

Richard M. Karp. 1960. A Note on the Application of Graph Theory to Digital Computer Programming. Information and Control 3, 2 (jun 1960), 179–190.

[35]

Caitlin Kelleher and Randy Pausch. 2005. Lowering the Barriers to Programming: A Taxonomy of Programming Environments and Languages for Novice Programmers. Comput. Surveys 37, 2 (jun 2005), 83–137.

Digital Library

[36]

Kostiantyn Kucher, Carita Paradis, Magnus Sahlgren, and Andreas Kerren. 2017. Active Learning and Visual Analytics for Stance Classification with ALVA. ACM Transactions on Interactive Intelligent Systems 7, 3, Article 14 (Oct. 2017), 31 pages.

Digital Library

[37]

Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, and Denis Charles. 2014. Structured Labeling for Facilitating Concept Evolution in Machine Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3075–3084.

Digital Library

[38]

Labelbox, Inc.2018. Labelbox. https://labelbox.com/

[39]

David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation Strategies for HCI Toolkit Research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1–17.

Digital Library

[40]

Fritz Lekschas, Brant Peterson, Daniel Haehn, Eric Ma, Nils Gehlenborg, and Hanspeter Pfister. 2020. Peax: Interactive Visual Pattern Search in Sequential Data Using Unsupervised Deep Representation Learning. Computer Graphics Forum 39, 3 (jun 2020), 167–179.

[41]

David D. Lewis and Jason Catlett. 1994. Heterogeneous Uncertainty Sampling for Supervised Learning. In Proceedings of the International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., 148–156.

[42]

Guoliang Li, Jiannan Wang, Yudian Zheng, and Michael J. Franklin. 2016. Crowdsourced Data Management: A Survey. IEEE Transactions on Knowledge and Data Engineering 28, 9 (sep 2016), 2296–2319.

Digital Library

[43]

Hongsen Liao, Li Chen, Yibo Song, and Hao Ming. 2016. Visualization-Based Active Learning for Video Annotation. IEEE Transactions on Multimedia 18, 11 (Nov. 2016), 2196–2205.

Digital Library

[44]

Tzuta Lin. 2015. LabelImg. https://github.com/tzutalin/labelImg

[45]

Dong Liu, Meng Wang, Xian-Sheng Hua, and Hong-Jiang Zhang. 2009. Smart Batch Tagging of Photo Albums. In Proceedings of the ACM International Conference on Multimedia. ACM, 809–812.

Digital Library

[46]

Shixia Liu, Changjian Chen, Yafeng Lu, Fangxin Ouyang, and Bin Wang. 2019. An Interactive Method to Improve Crowdsourced Annotations. IEEE Transactions on Visualization and Computer Graphics 25, 1 (Jan. 2019), 235–245.

Digital Library

[47]

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. In Proceedings of the Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 142–150.

[48]

Gonzalo Gabriel Méndez, Miguel A. Nacenta, and Sebastien Vandenheste. 2016. iVoLVER: Interactive Visual Language for Visualization Extraction and Reconstruction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 4073–4085.

Digital Library

[49]

Microsoft. 2017. VoTT: Visual Object Tagging Tool. https://github.com/microsoft/VoTT

[50]

Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text Annotation Tool for Human. https://github.com/doccano/doccano

[51]

Curtis Northcutt, Lu Jiang, and Isaac Chuang. 2021. Confident Learning: Estimating Uncertainty in Dataset Labels. Journal of Artificial Intelligence Research 70 (may 2021), 1373–1411.

[52]

Allard Oelen, Markus Stocker, and Sören Auer. 2021. Crowdsourcing Scholarly Discourse Annotations. In Proceedings of the International Conference on Intelligent User Interfaces. ACM, 464–474.

Digital Library

[53]

Jose Gustavo S. Paiva, William Robson Schwartz, Helio Pedrini, and Rosane Minghim. 2015. An Approach to Supporting Incremental Visual Data Classification. IEEE Transactions on Visualization and Computer Graphics 21, 1 (Jan. 2015), 4–17.

[54]

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based on public domain audio books. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 5206–5210.

[55]

Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid Training Data Creation with Weak Supervision. Proceedings of the VLDB Endowment 11, 3 (Nov. 2017), 269–282.

Digital Library

[56]

Donghao Ren, Tobias Hollerer, and Xiaoru Yuan. 2014. iVisDesigner: Expressive Interactive Design of Information Visualizations. IEEE Transactions on Visualization and Computer Graphics 20, 12 (Dec. 2014), 2092–2101.

[57]

Tim Rietz and Alexander Maedche. 2021. Cody: An AI-Based System to Semi-Automate Coding for Qualitative Research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Article 394, 14 pages.

Digital Library

[58]

Olga Russakovsky, Li-Jia Li, and Li Fei-Fei. 2015. Best of both worlds: Human-machine collaboration for object annotation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2121–2131.

[59]

Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2008. LabelMe: A Database and Web-Based Tool for Image Annotation. International Journal of Computer Vision 77, 1-3 (2008), 157–173.

Digital Library

[60]

Christian Schüldt, Ivan Laptev, and Barbara Caputo. 2004. Recognizing Human Actions: A Local SVM Approach. In Proceedings of the International Conference on Pattern Recognition, Vol. 3. IEEE, 32–36.

[61]

Burr Settles. 2009. Active Learning Literature Survey. Technical Report. University of Wisconsin–Madison.

[62]

Burr Settles. 2011. Closing the Loop: Fast, Interactive Semi-supervised Annotation with Queries on Features and Instances. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1467–1478.

[63]

Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. 2019. Annotating Objects and Relations in User-Generated Videos. In Proceedings of the International Conference on Multimedia Retrieval. ACM, 279–287.

Digital Library

[64]

Bongwon Suh and Benjamin B. Bederson. 2007. Semi-Automatic Photo Annotation Strategies Using Event Based Clustering and Clothing Based Person Recognition. Interacting with Computers 19, 4 (July 2007), 524–544.

Digital Library

[65]

Jinhui Tang, Qiang Chen, Meng Wang, Shuicheng Yan, Tat-Seng Chua, and Ramesh Jain. 2013. Towards Optimizing Human Labeling for Interactive Image Tagging. ACM Transactions on Multimedia Computing, Communications, and Applications 9, 4, Article 29 (Aug. 2013), 18 pages.

Digital Library

[66]

Yuandong Tian, Wei Liu, Rong Xiao, Fang Wen, and Xiaoou Tang. 2007. A Face Annotation Framework with Partial Clustering and Interactive Labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–8.

[67]

Maxim Tkachenko, Mikhail Malyuk, Nikita Shevchenko, Andrey Holmanyuk, and Nikolai Liubimov. 2020. Label Studio: Data labeling software. https://github.com/heartexlabs/label-studio

[68]

Luis von Ahn and Laura Dabbish. 2004. Labeling Images with a Computer Game. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 319–326.

Digital Library

[69]

Luis von Ahn and Laura Dabbish. 2008. Designing Games with a Purpose. Commun. ACM 51, 8 (Aug. 2008), 58–67.

Digital Library

[70]

Luis von Ahn, Ruoran Liu, and Manuel Blum. 2006. Peekaboom: A Game for Locating Objects in Images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 55–64.

Digital Library

[71]

Luis von Ahn, Benjamin Maurer, Colin McMillen, David Abraham, and Manuel Blum. 2008. reCAPTCHA: Human-Based Character Recognition via Web Security Measures. Science 321, 5895 (aug 2008), 1465–1468.

[72]

Guotai Wang, Maria A. Zuluaga, Wenqi Li, Rosalind Pratt, Premal A. Patel, Michael Aertsen, Tom Doel, Anna L. David, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren. 2019. DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 7 (jul 2019), 1559–1572.

[73]

Meng Wang and Xian-Sheng Hua. 2011. Active Learning in Multimedia Annotation and Retrieval: A Survey. ACM Transactions on Intelligent Systems and Technology 2, 2, Article 10 (Feb. 2011), 21 pages.

Digital Library

[74]

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A Deep Representation for Volumetric Shapes. In IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1912–1920.

[75]

Shouxing Xiang, Xi Ye, Jiazhi Xia, Jing Wu, Yang Chen, and Shixia Liu. 2019. Interactive Correction of Mislabeled Training Data. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology. IEEE, 57–68.

[76]

Zuobing Xu, Ram Akella, and Yi Zhang. 2007. Incorporating Diversity and Density in Active Learning for Relevance Feedback. In Proceedings of the European Conference on IR Research. Springer-Verlag, 246–257.

[77]

Yang Yang, Bin B. Zhu, Rui Guo, Linjun Yang, Shipeng Li, and Nenghai Yu. 2008. A Comprehensive Human Computation Framework: With Application to Image Labeling. In Proceedings of the ACM International Conference on Multimedia. ACM, 479–488.

Digital Library

[78]

Hao Ye, Weiyuan Shao, Hong Wang, Jianqi Ma, Li Wang, Yingbin Zheng, and Xiangyang Xue. 2016. Face Recognition via Active Annotation and Learning. In Proceedings of the ACM International Conference on Multimedia. ACM, 1058–1062.

Digital Library

[79]

Bowen Yu and Claudio T. Silva. 2017. VisFlow - Web-based Visualization Framework for Tabular Data with a Subset Flow Model. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 251–260.

Digital Library

[80]

Jan Zahálka and Marcel Worring. 2014. Towards Interactive, Intelligent, and Integrated Multimedia Analytics. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology. IEEE, 3–12.

[81]

Jan Zahálka, Marcel Worring, and Jarke J. van Wijk. 2020. II-20: Intelligent and pragmatic analytic categorization of image collections. IEEE Transactions on Visualization and Computer Graphics 27, 2(2020), 422–431.

[82]

Lishi Zhang, Chenghan Fu, and Jia Li. 2018. Collaborative Annotation of Semantic Objects in Images with Multi-Granularity Supervisions. In Proceedings of the ACM International Conference on Multimedia. ACM, 474–482.

Digital Library

[83]

Tianye Zhang, Haozhe Feng, Wei Chen, Zexian Chen, Wenting Zheng, Xiao-Nan Luo, Wenqi Huang, and Anthony K. H. Tung. 2021. ChartNavigator: An Interactive Pattern Identification and Annotation Framework for Charts. IEEE Transactions on Knowledge and Data Engineering (2021), 1–1.

[84]

Yexun Zhang, Wenbin Cai, Wenquan Wang, and Ya Zhang. 2017. Stopping Criterion for Active Learning with Model Stability. ACM Transactions on Intelligent Systems and Technology 9, 2, Article 19 (Oct. 2017), 26 pages.

[85]

Yu Zhang, Bob Coecke, and Min Chen. 2021. MI3: Machine-Initiated Intelligent Interaction for Interactive Classification and Data Reconstruction. ACM Transactions on Interactive Intelligent Systems 11, 3–4, Article 18 (Aug. 2021), 34 pages.

Digital Library

[86]

Yu Zhang, Martijn Tennekes, Tim de Jong, Lyana Curier, Bob Coecke, and Min Chen. 2021. Using Simulation to Aid the Design and Optimization of Intelligent User Interfaces for Quality Assurance Processes in Machine Learning. arXiv:2104.01129 [cs.HC]

[87]

Jingbo Zhu, Huizhen Wang, Eduard Hovy, and Matthew Ma. 2010. Confidence-Based Stopping Criteria for Active Learning for Data Annotation. ACM Transactions on Speech and Language Processing 6, 3, Article 3 (April 2010), 24 pages.

Digital Library

[88]

Xingquan Zhu, Xindong Wu, and Qijun Chen. 2003. Eliminating Class Noise in Large Datasets. In Proceedings of the International Conference on Machine Learning. AAAI Press, 920–927.

Cited By

Narushynska OTeslyuk VDoroshenko AArzubov M(2024)Data Sorting Influence on Short Text Manual Labeling Quality for Hierarchical ClassificationBig Data and Cognitive Computing10.3390/bdcc80400418:4(41)Online publication date: 7-Apr-2024
https://doi.org/10.3390/bdcc8040041
Lyu HBai YLiang XDas UShi CGong LLi YSun MGe MMa X(2024)FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers’ Preference ElicitationProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645145(344-369)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645145
Zhang YTennekes MDe Jong TCurier LCoecke BChen M(2024)Simulation-based Optimization of User Interfaces for Quality-assuring Machine Learning Model PredictionsACM Transactions on Interactive Intelligent Systems10.1145/359455214:1(1-32)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1145/3594552
Show More Cited By

Index Terms

OneLabeler: A Flexible System for Building Data Labeling Tools
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information systems applications
    1. Multimedia information systems

Recommendations

Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface
IUI '21: Proceedings of the 26th International Conference on Intelligent User Interfaces

Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has ...
A Cluster-then-label Approach for Few-shot Learning with Application to Automatic Image Data Labeling
Few-shot learning (FSL) aims at learning to generalize from only a small number of labeled examples for a given target task. Most current state-of-the-art FSL methods typically have two limitations. First, they usually require access to a source dataset (...
LabelUX! Guidelines to support software engineers to design data labeling systems
SBQS '21: Proceedings of the XX Brazilian Symposium on Software Quality

The demand for systems using artificial intelligence has substantially boosted in recent times, especially with Machine Learning (ML) techniques. Systems that use ML supervision techniques need representative and correctly categorized data to ensure ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

10459 pages

ISBN:9781450391573

DOI:10.1145/3491102

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA
,
Steven Drucker
Microsoft Research, USA
,
Julie Williamson
University of Glasgow, UK
,
Koji Yatani
University of Tokyo, Japan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Shanghai Sailing Program
Shanghai Science and Technology Commission
Shanghai Municipal Science and Technology Major Project

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
973
Total Downloads

Downloads (Last 12 months)174
Downloads (Last 6 weeks)22

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Narushynska OTeslyuk VDoroshenko AArzubov M(2024)Data Sorting Influence on Short Text Manual Labeling Quality for Hierarchical ClassificationBig Data and Cognitive Computing10.3390/bdcc80400418:4(41)Online publication date: 7-Apr-2024
https://doi.org/10.3390/bdcc8040041
Lyu HBai YLiang XDas UShi CGong LLi YSun MGe MMa X(2024)FARPLS: A Feature-Augmented Robot Trajectory Preference Labeling System to Assist Human Labelers’ Preference ElicitationProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645145(344-369)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645145
Zhang YTennekes MDe Jong TCurier LCoecke BChen M(2024)Simulation-based Optimization of User Interfaces for Quality-assuring Machine Learning Model PredictionsACM Transactions on Interactive Intelligent Systems10.1145/359455214:1(1-32)Online publication date: 9-Jan-2024
https://dl.acm.org/doi/10.1145/3594552
Shao HYuan X(2024)CataAnno: An Ancient Catalog Annotator for Annotation Cleaning by RecommendationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345637931:1(404-414)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3456379
Xie GFu HXue HLiu BXu XLi XSun Z(2024)M3-GMN: A Multi-environment, Multi-LiDAR, Multi-task dataset for Grid Map based Navigation2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS58592.2024.10802553(13213-13220)Online publication date: 14-Oct-2024
https://doi.org/10.1109/IROS58592.2024.10802553
Gomez MRuipérez-Valiente JGarcía Clemente F(2024)Optimizing multimedia and gameplay data labeling: A web-based tool for Game-Based AssessmentSoftwareX10.1016/j.softx.2024.10176327(101763)Online publication date: Sep-2024
https://doi.org/10.1016/j.softx.2024.101763
Nigatu HPickoff-White LCanny JChasins S(2023)Co-Designing for Transparency: Lessons from Building a Document Organization Tool in the Criminal Justice DomainProceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency10.1145/3593013.3594093(1463-1478)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3593013.3594093
Zhang YJiang RXie LZhao YLiu CDing TChen SYuan X(2023)OldVisOnline: Curating a Dataset of Historical VisualizationsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332690830:1(551-561)Online publication date: 24-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3326908
He JWang XWong KHuang XChen CChen ZWang FZhu MQu H(2023): A Visual Analytics Approach for Interactive Video ProgrammingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332658630:1(87-97)Online publication date: 23-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3326586
Zhang XXuan XDima ASexton TMa K(2023)LabelVizier: Interactive Validation and Relabeling for Technical Text Annotations2023 IEEE 16th Pacific Visualization Symposium (PacificVis)10.1109/PacificVis56936.2023.00026(167-176)Online publication date: Apr-2023
https://doi.org/10.1109/PacificVis56936.2023.00026
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents