research-article

AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning

Authors:

Soroush Ghorashi,

Steven Drucker,

Patrice SimardAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 10, Issue 1

Article No.: 7, Pages 1 - 38

https://doi.org/10.1145/3241379

Published: 09 August 2019 Publication History

Abstract

When building a classifier in interactive machine learning (iML), human knowledge about the target class can be a powerful reference to make the classifier robust to unseen items. The main challenge lies in finding unlabeled items that can either help discover or refine concepts for which the current classifier has no corresponding features (i.e., it has feature blindness). Yet it is unrealistic to ask humans to come up with an exhaustive list of items, especially for rare concepts that are hard to recall. This article presents AnchorViz, an interactive visualization that facilitates the discovery of prediction errors and previously unseen concepts through human-driven semantic data exploration. By creating example-based or dictionary-based anchors representing concepts, users create a topology that (a) spreads data based on their similarity to the concepts and (b) surfaces the prediction and label inconsistencies between data points that are semantically related. Once such inconsistencies and errors are discovered, users can encode the new information as labels or features and interact with the retrained classifier to validate their actions in an iterative loop. We evaluated AnchorViz through two user studies. Our results show that AnchorViz helps users discover more prediction errors than stratified random and uncertainty sampling methods. Furthermore, during the beginning stages of a training task, an iML tool with AnchorViz can help users build classifiers comparable to the ones built with the same tool with uncertainty sampling and keyword search, but with fewer labels and more generalizable features. We discuss exploration strategies observed during the two studies and how AnchorViz supports discovering, labeling, and refining of concepts through a sensemaking loop.

References

[1]

Charu C. Aggarwal and ChengXiang Zhai. 2012. Mining Text Data. Springer Science 8 Business Media.

Digital Library

[2]

Jae-wook Ahn and Peter Brusilovsky. 2009. Adaptive visualization of search results: Bringing user models to visual analytics. Information Visualization 8, 3 (2009), 167--179.

Digital Library

[3]

Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. AI Magazine 35, 4 (2014), 105--120.

Digital Library

[4]

Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015. Modeltracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 337--346.

Digital Library

[5]

Saleema Amershi, James Fogarty, and Daniel Weld. 2012. Regroup: Interactive machine learning for on-demand group creation in social networks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 21--30.

Digital Library

[6]

Joshua Attenberg, Panos Ipeirotis, and Foster Provost. 2015. Beat the machine: Challenging humans to find a predictive model’s “unknown unknowns”. Journal of Data and Information Quality (JDIQ) 6, 1 (2015), 1.

Digital Library

[7]

Josh Attenberg and Foster Provost. 2010. Why label when you can search?: Alternatives to active learning for applying human resources to build classification models under extreme class imbalance. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 423--432.

Digital Library

[8]

Michael Brooks, Saleema Amershi, Bongshin Lee, Steven M. Drucker, Ashish Kapoor, and Patrice Simard. 2015. FeatureInsight: Visual support for error-driven feature ideation in text classification. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST’15). IEEE, 105--112.

[9]

Mackinlay Card. 1999. Readings in Information Visualization: Using Vision to Think. Morgan-Kaufmann.

Digital Library

[10]

Duen Horng Chau, Aniket Kittur, Jason I. Hong, and Christos Faloutsos. 2011. Apolo: Making sense of large network data by combining rich user interaction and machine learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 167--176.

Digital Library

[11]

Nan-Chen Chen, Jina Suh, Johan Verwey, Gonzalo Ramos, Steven Drucker, and Patrice Simard. 2018. AnchorViz: Facilitating classifier error discovery through interactive semantic data exploration. In Proceedings of the 23rd International Conference on Intelligent User Interfaces. ACM, 269--280.

Digital Library

[12]

Justin Cheng and Michael S. Bernstein. 2015. Flock: Hybrid crowd-machine learning classifiers. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work 8 Social Computing. ACM, 600--611.

Digital Library

[13]

Jason Chuang, Sonal Gupta, Christopher Manning, and Jeffrey Heer. 2013. Topic model diagnostics: Assessing domain relevance via topical alignment. In Proceedings of the International Conference on Machine Learning. 612--620.

Digital Library

[14]

Aron Culotta, Trausti Kristjansson, Andrew McCallum, and Paul Viola. 2006. Corrective feedback and persistent learning for information extraction. Artificial Intelligence 170, 14–15 (2006), 1101--1122.

Digital Library

[15]

Pedro Domingos. 2012. A few useful things to know about machine learning. Commun. ACM 55, 10 (2012), 78--87.

Digital Library

[16]

A. Endert, W. Ribarsky, C. Turkay, B. L. Wong, Ian Nabney, I. Díaz Blanco, and F. Rossi. 2017. The state of the art in integrating machine learning into visual analytics. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 458--486.

[17]

Jerry Alan Fails and Dan R. Olsen, Jr. 2003. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (IUI’03). ACM, New York, 39--45.

[18]

James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive concept learning in image search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 29--38.

Digital Library

[19]

Thomas M. J. Fruchterman and Edward M. Reingold. 1991. Graph drawing by force-directed placement. Software: Practice and Experience 21, 11 (1991), 1129--1164.

Digital Library

[20]

Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 145--154.

Digital Library

[21]

Florian Heimerl, Charles Jochim, Steffen Koch, and Thomas Ertl. 2012. FeatureForge: A novel tool for visually supported feature engineering and corpus revision. In COLING.

[22]

Patrick E Hoffman. {n.d.}. Table Visualizations: A Formal Model and Its Applications. Ph.D. Dissertation. University of Massachusetts. Lowell.

Digital Library

[23]

Rong Hu, Sarah Jane Delany, and Brian Mac Namee. 2010. EGAL: Exploration guided active learning for TCBR. In Proceedings of the International Conference on Case-Based Reasoning. Springer, 156--170.

Digital Library

[24]

Xinran Hu, Lauren Bradel, Dipayan Maiti, Leanna House, and Chris North. 2013. Semantics of directly manipulating spatializations. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2052--2059.

Digital Library

[25]

Camille Jandot, Patrice Simard, Max Chickering, David Grangier, and Jina Suh. 2016. Interactive semantic featuring for text classification. arXiv preprint arXiv:1606.07545 (2016).

[26]

Ian Jolliffe. 2011. Principal component analysis. In International Encyclopedia of Statistical Science. Springer, 1094--1096.

[27]

Hannah Kim, Jaegul Choo, Haesun Park, and Alex Endert. 2016. InterAxis: Steering scatterplot axes via observation-level interaction. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 131--140.

Digital Library

[28]

Josua Krause, Adam Perer, and Enrico Bertini. 2014. INFUSE: Interactive feature selection for predictive modeling of high dimensional data. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1614--1623.

[29]

Josua Walter Hugo Krause. 2018. Using Visual Analytics to Explain Black-Box Machine Learning. Ph.D. Dissertation. New York University Tandon School of Engineering.

[30]

Todd Kulesza, Saleema Amershi, Rich Caruana, Danyel Fisher, and Denis Charles. 2014. Structured labeling for facilitating concept evolution in machine learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3075--3084.

Digital Library

[31]

Todd Kulesza, Simone Stumpf, Weng-Keen Wong, Margaret M. Burnett, Stephen Perona, Andrew Ko, and Ian Oberst. 2011. Why-oriented end-user debugging of naive Bayes text classification. ACM Transactions on Interactive Intelligent Systems (TiiS) 1, 1 (2011), 2.

Digital Library

[32]

Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Eric Horvitz. 2017. Identifying unknown unknowns in the open world: Representations and policies for guided exploration. In Proceedings of AAAI. 2124--2132.

Digital Library

[33]

Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Proceedings of the 12th International Conference on Machine Learning, Vol. 10. 331--339.

Digital Library

[34]

Hanseung Lee, Jaeyeon Kihm, Jaegul Choo, John Stasko, and Haesun Park. 2012. iVisClustering: An interactive visual document clustering via topic modeling. Computer Graphics Forum 31, 3pt3 (June 2012), 1155--1164. 00041

Digital Library

[35]

David D. Lewis and Jason Catlett. 1994. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of the 11th International Conference on Machine Learning. 148--156.

Digital Library

[36]

David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Springer-Verlag New York, Inc., 3--12.

[37]

Shusen Liu, Dan Maljovec, Bei Wang, Peer-Timo Bremer, and Valerio Pascucci. 2015. Visualizing high-dimensional data: Advances in the past decade. IEEE Transactions on Visualization and Computer Graphics 23, 3 (2017), 1249–1268.

Digital Library

[38]

Shixia Liu, Xiting Wang, Mengchen Liu, and Jun Zhu. 2017. Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics 1, 1 (2017), 48--56.

[39]

Shixia Liu, Jiannan Xiao, Junlin Liu, Xiting Wang, Jing Wu, and Jun Zhu. 2018. Visual diagnosis of tree boosting methods. IEEE Transactions on Visualization 8 Computer Graphics 1 (2018), 1--1.

Digital Library

[40]

Yafeng Lu, Rolando Garcia, Brett Hansen, Michael Gleicher, and Ross Maciejewski. 2017. The state-of-the-art in predictive visual analytics. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 539--562.

Digital Library

[41]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, Nov (2008), 2579--2605.

[42]

Brian Mac Namee, Rong Hu, and Sarah Jane Delany. 2010. Inside the selection box: Visualising active learning selection strategies. In Proceedings of The Challenges of Data Visualization Neural Information Processing Systems (NIPS) Workshop. Dublin Institute of Technology.

[43]

Christopher D. Manning and Hinrich Schütze. 1999. Foundations of Statistical Natural Language Processing. MIT Press.

Digital Library

[44]

Christopher Meek. 2016. A characterization of prediction errors. CoRR abs/1611.05955 (2016). http://arxiv.org/abs/1611.05955

[45]

Gregory Murphy. 2004. The Big Book of Concepts. MIT Press.

[46]

R. M. Nosofsky. 1986. Attention, similarity, and the identification-categorization relationship. Journal of Experimental Psychology. General 115 1 (1986), 39--61.

[47]

Kai A. Olsen, James G. Williams, Kenneth M. Sochats, and Stephen C. Hirtle. 1992. Ideation through visualization: The VIBE system. Multimedia Review 3 (1992), 48--48.

[48]

Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of the International Conference on Intelligence Analysis, Vol. 5. 2--4.

[49]

Hema Raghavan, Omid Madani, and Rosie Jones. 2005. InterActive feature selection. In Proceedings of IJCAI, Vol. 5. 841--846.

Digital Library

[50]

D. Ren, S. Amershi, B. Lee, J. Suh, and J. D. Williams. 2017. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 61--70. 00000.

Digital Library

[51]

Eleanor Rosch and Carolyn B. Mervis. 1975. Family resemblances: Studies in the internal structure of categories. Cognitive Psychology 7, 4 (1975), 573--605.

[52]

Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993. The cost structure of sensemaking. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems (CHI’93). ACM, New York, 269--276.

Digital Library

[53]

John W. Sammon. 1969. A nonlinear mapping for data structure analysis. IEEE Transactions on Computers 100, 5 (1969), 401--409.

Digital Library

[54]

Sam Scott and Stan Matwin. 1999. Feature engineering for text classification. In Proceedings of ICML, Vol. 99. 379--388.

Digital Library

[55]

Burr Settles. 2011. Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1467--1478.

Digital Library

[56]

Burr Settles. 2012. Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 6, 1 (2012), 1--114.

Digital Library

[57]

John Sharko, Georges Grinstein, and Kenneth A Marx. 2008. Vectorized radviz and its application to multiple cluster datasets. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008).

Digital Library

[58]

Ben Shneiderman. 1992. Tree visualization with tree-maps: 2-d space-filling approach. ACM Trans. Graph. 11, 1 (Jan. 1992), 92--99.

Digital Library

[59]

Edward E. Smith and Douglas L. Medin. 1981. Categories and Concepts. Vol. 9. Harvard University Press, Cambridge, MA.

[60]

Robert R. Sokal. 1958. A statistical method for evaluating systematic relationship. University of Kansas Science Bulletin 28 (1958), 1409--1438.

[61]

Ji Soo Yi, Rachel Melton, John Stasko, and Julie A. Jacko. 2005. Dust 8 magnet: Multivariate information visualization using a magnet metaphor. Information Visualization 4, 4 (2005), 239--256.

Digital Library

[62]

Simone Stumpf, Vidya Rajaram, Lida Li, Weng-Keen Wong, Margaret Burnett, Thomas Dietterich, Erin Sullivan, and Jonathan Herlocker. 2009. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies 67, 8 (2009), 639--662.

Digital Library

[63]

Maite Taboada, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. Computational Linguistics 37, 2 (2011), 267--307.

Digital Library

[64]

Justin Talbot, Bongshin Lee, Ashish Kapoor, and Desney S. Tan. 2009. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’09). ACM, New York, 1283--1292. 00097.

Digital Library

[65]

Stef Van Den Elzen and Jarke J van Wijk. 2011. Baobabview: Interactive construction and analysis of decision trees. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST’11). IEEE, 151--160.

[66]

Alfredo Vellido Alcacena, José David Martín, Fabrice Rossi, and Paulo J. G. Lisboa. 2011. Seeing is believing: The importance of visualization in real-world machine learning applications. In Proceedings of the 19th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2011: Bruges, Belgium, April 27-28-29, 2011. 219--226.

[67]

Byron C. Wallace, Kevin Small, Carla E. Brodley, Joseph Lau, and Thomas A. Trikalinos. 2012. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium. ACM, 819--824.

Digital Library

[68]

Malcolm Ware, Eibe Frank, Geoffrey Holmes, Mark Hall, and Ian H. Witten. 2001. Interactive machine learning: Letting users build classifiers. International Journal of Human-Computer Studies 55, 3 (2001), 281--292.

Digital Library

[69]

Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In ICML, Vol. 97. 412--420.

Digital Library

[70]

J. Zhang, Y. Wang, P. Molino, L. Li, and D. S. Ebert. 2018. Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Transactions on Visualization and Computer Graphics (2018).

Cited By

Szymanski MOoge JDe Croon RVanden Abeele VVerbert K(2024)Feedback, Control, or Explanations? Supporting Teachers With Steerable Distractor-Generating AIProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636933(690-700)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636933
Song HRamos GBodik P(2024)Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes2024 IEEE 17th Pacific Visualization Conference (PacificVis)10.1109/PacificVis60374.2024.00043(307-312)Online publication date: 23-Apr-2024
https://doi.org/10.1109/PacificVis60374.2024.00043
Kawabe WNakao YShitara ASugano Y(2024)Technical Understanding from Interactive Machine Learning Experience: a Study Through a Public Event for Science Museum VisitorsInteracting with Computers10.1093/iwc/iwae00736:3(155-171)Online publication date: 12-Mar-2024
https://doi.org/10.1093/iwc/iwae007
Show More Cited By

Index Terms

AnchorViz: Facilitating Semantic Data Exploration and Concept Discovery for Interactive Machine Learning
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

AnchorViz: Facilitating Classifier Error Discovery through Interactive Semantic Data Exploration
IUI '18: Proceedings of the 23rd International Conference on Intelligent User Interfaces

When building a classifier in interactive machine learning, human knowledge about the target class can be a powerful reference to make the classifier robust to unseen items. The main challenge lies in finding unlabeled items that can either help discover ...
Transductive Multilabel Learning via Label Set Propagation

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image ...
Learning Instance Weighted Naive Bayes from labeled and unlabeled data

In real-world data mining applications, it is often the case that unlabeled instances are abundant, while available labeled instances are very limited. Thus, semi-supervised learning, which attempts to benefit from large amount of unlabeled data ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 10, Issue 1

Special Issue on IUI 2018

March 2020

347 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3352585

Editor:
Michelle X. Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 August 2019

Accepted: 01 November 2018

Revised: 01 September 2018

Received: 01 May 2018

Published in TIIS Volume 10, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
626
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)7

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Szymanski MOoge JDe Croon RVanden Abeele VVerbert K(2024)Feedback, Control, or Explanations? Supporting Teachers With Steerable Distractor-Generating AIProceedings of the 14th Learning Analytics and Knowledge Conference10.1145/3636555.3636933(690-700)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3636555.3636933
Song HRamos GBodik P(2024)Evaluating how interactive visualizations can assist in finding samples where and how computer vision models make mistakes2024 IEEE 17th Pacific Visualization Conference (PacificVis)10.1109/PacificVis60374.2024.00043(307-312)Online publication date: 23-Apr-2024
https://doi.org/10.1109/PacificVis60374.2024.00043
Kawabe WNakao YShitara ASugano Y(2024)Technical Understanding from Interactive Machine Learning Experience: a Study Through a Public Event for Science Museum VisitorsInteracting with Computers10.1093/iwc/iwae00736:3(155-171)Online publication date: 12-Mar-2024
https://doi.org/10.1093/iwc/iwae007
Ahn YLin YXu PDai Z(2023)ESCAPE: Countering Systematic Errors from Machine’s Blind Spots via Interactive Visual AnalysisProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581373(1-16)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581373
Deng WGuo BDevrio AShen HEslami MHolstein K(2023)Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry PracticeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581026(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581026
Huang ZWitschard DKucher KKerren A(2023)VA + Embeddings STAR: A State‐of‐the‐Art Report on the Use of Embeddings in Visual AnalyticsComputer Graphics Forum10.1111/cgf.1485942:3(539-571)Online publication date: 27-Jun-2023
https://doi.org/10.1111/cgf.14859
DeVos ADhabalia AShen HHolstein KEslami M(2022)Toward User-Driven Algorithm Auditing: Investigating users’ strategies for uncovering harmful algorithmic behaviorProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517441(1-19)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517441
Mosqueira-Rey EPereira EAlonso-Ríos DBobes-Bascarán JHong JBures MPark JCerny T(2022)A classification and review of tools for developing and interacting with machine learning systemsProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507310(1092-1101)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3477314.3507310
Bhardwaj AYang JCudre-Mauroux P(2022)Human-in-the-Loop Rule Discovery for Micropost Event DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.3208345(1-12)Online publication date: 2022
https://doi.org/10.1109/TKDE.2022.3208345
Mosqueira-Rey EHernández-Pereira EAlonso-Ríos DBobes-Bascarán JFernández-Leal Á(2022)Human-in-the-loop machine learning: a state of the artArtificial Intelligence Review10.1007/s10462-022-10246-w56:4(3005-3054)Online publication date: 17-Aug-2022
https://dl.acm.org/doi/10.1007/s10462-022-10246-w
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents