skip to main content
research-article

MI3: Machine-initiated Intelligent Interaction for Interactive Classification and Data Reconstruction

Published: 03 September 2021 Publication History

Abstract

In many applications, while machine learning (ML) can be used to derive algorithmic models to aid decision processes, it is often difficult to learn a precise model when the number of similar data points is limited. One example of such applications is data reconstruction from historical visualizations, many of which encode precious data, but their numerical records are lost. On the one hand, there is not enough similar data for training an ML model. On the other hand, manual reconstruction of the data is both tedious and arduous. Hence, a desirable approach is to train an ML model dynamically using interactive classification, and hopefully, after some training, the model can complete the data reconstruction tasks with less human interference. For this approach to be effective, the number of annotated data objects used for training the ML model should be as small as possible, while the number of data objects to be reconstructed automatically should be as large as possible. In this article, we present a novel technique for the machine to initiate intelligent interactions to reduce the user’s interaction cost in interactive classification tasks. The technique of machine-initiated intelligent interaction (MI3) builds on a generic framework featuring active sampling and default labeling. To demonstrate the MI3 approach, we use the well-known cholera map visualization by John Snow as an example, as it features three instances of MI3 pipelines. The experiment has confirmed the merits of the MI3 approach.

References

[1]
Alfie Abdul-Rahman, Gleen Roe, Mark Olsen, Clovis Gladstone, Richard Whaling, Nicholas Cronk, Robert Morrissey, and Min Chen. 2017. Constructive visual analytics for text similarity detection. Comput. Graph. For. 36, 1 (2017), 237–248.
[2]
Rabah A. Al-Zaidy and Clyde Lee Giles. 2017. A machine learning approach for semantic structuring of scientific charts in scholarly documents. In Proceedings of the AAAI Conference on Innovative Applications (IAAI’17). AAAI Press, 4644–4649.
[3]
Mykhaylo Andriluka, Jasper R. R. Uijlings, and Vittorio Ferrari. 2018. Fluid annotation: A human-machine collaboration interface for full image annotation. In Proceedings of the ACM International Conference on Multimedia (MM’18). ACM, 1957–1966.
[4]
Jürgen Bernard, Christian Ritter, David Sessler, Matthias Zeppelzauer, Jörn Kohlhammer, and Dieter Fellner. 2017. Visual-interactive similarity search for complex objects by example of soccer player analysis. In Proceedings of the International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. INSTICC, SciTePress, 75–87.
[5]
Jürgen Bernard, Matthias Zeppelzauer, Michael Sedlmair, and Wolfgang Aigner. 2018. VIAL: A unified process for visual interactive labeling. Vis. Comput. 34, 9 (2018), 1189–1207.
[6]
Jacques Bertillon. 1886. Paris 1886 Population par Groupes d’Âges et par Quartier. George Masson, Paris.
[7]
Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the Conference on Computational Learning Theory (COLT’98). ACM, 92–100.
[8]
Simon Bovet and Jean Bovet. 2006. GraphClick. Arizona Software.
[9]
Klaus Brinker. 2003. Incorporating diversity in active learning with support vector machines. In Proceedings of the International Conference on Machine Learning (ICML’03). AAAI Press, 59–66.
[10]
Nicholas J. Bryan, Gautham J. Mysore, and Ge Wang. 2014. ISSE: An interactive source separation editor. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’14). ACM, 257–266.
[11]
Stuart K. Card, Jock D. Mackinlay, and Ben Shneiderman (Eds.). 1999. Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann Publishers Inc.
[12]
Stuart K. Card, Thomas P. Moran, and Allen Newell. 1980. The keystroke-level model for user performance time with interactive systems. Commun. ACM 23, 7 (July 1980), 396–410.
[13]
Stuart K. Card, Thomas P. Moran, and Allen Newell. 1983. The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates.
[14]
Arthur Cayley. 1890. The Collected Mathematical Papers of Arthur Cayley, Volume 3. Cambridge University Press.
[15]
Olivier Chapelle, Jason Weston, and Bernhard Schölkopf. 2002. Cluster kernels for semi-supervised learning. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS’02). The MIT Press, 601–608.
[16]
Émile Cheysson. 1906. Circulation sur les Routes Nationales, de 1851 à 1894. Imprimerie Nationale.
[17]
Ed Huai-Hsin Chi. 2000. A taxonomy of visualization techniques using the data state reference model. In Proceedings of the IEEE Symposium on Information Visualization (INFOVIS’00). IEEE, 69–75.
[18]
Ed Huai-Hsin Chi and John Thomas Riedl. 1998. An operator interaction framework for visualization systems. In Proceedings of the IEEE Symposium on Information Visualization (INFOVIS’98). IEEE, 63–70.
[19]
Jingyu Cui, Fang Wen, Rong Xiao, Yuandong Tian, and Xiaoou Tang. 2007. EasyAlbum: An interactive photo annotation system based on face clustering and re-ranking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’07). ACM, 367–376.
[20]
Adolphe d’Angeville. 1836. Statistique de la Population Française, Population Habitans par myriametres carries en 1831. Fred Dufour, Bourg.
[21]
Jerry Alan Fails and Dan R. Olsen. 2003. Interactive machine learning. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’03). ACM, 39–45.
[22]
Usama Fayyad, Gregory Piatetsky-Shapiro, and Padhraic Smyth. 1996. From data mining to knowledge discovery in databases. AI Mag. 17, 3 (1996), 37–54.
[23]
James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: Interactive concept learning in image search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, 29–38.
[24]
Michael Friendly and Daniel J. Denis. 2001. Milestones in the History of Thematic Cartography, Statistical Graphics, and Data Visualization. Retrieved on 31 May, 2020 from http://www.datavis.ca/milestones.
[25]
Jinglun Gao, Yin Zhou, and Kenneth E. Barner. 2012. View: Visual information extraction widget for improving chart images accessibility. In Proceedings of the IEEE International Conference on Image Processing (ICIP’12). IEEE, 2865–2868.
[26]
Melinda T. Gervasio, Michael D. Moffitt, Martha E. Pollack, Joseph M. Taylor, and Tomas E. Uribe. 2005. Active preference learning for personalized calendar scheduling assistance. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’05). ACM, 90–97.
[27]
Arnd Gross, Sibylle Schirm, and Markus Scholz. 2014. Ycasd – A tool for capturing and scaling data from graphical representations. BMC Bioinf. 15, 1 (2014), 219.
[28]
Yuhong Guo and Dale Schuurmans. 2007. Discriminative batch mode active learning. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS’07). Curran Associates Inc., 593–600.
[29]
Zicheng Guo and Richard W. Hall. 1989. Parallel thinning with two-subiteration algorithms. Commun. ACM 32, 3 (1989), 359–373.
[30]
Florian Heimerl, Steffen Koch, Harald Bosch, and Thomas Ertl. 2012. Visual classifier training for text document retrieval. IEEE Trans. Vis. Comput. Graph. 18, 12 (Dec. 2012), 2839–2848.
[31]
Benjamin Höferlin, Rudolf Netzel, Markus Höferlin, Daniel Weiskopf, and Gunther Heidemann. 2012. Inter-active learning of ad-hoc classifiers for video visual analytics. In Proceedings of the IEEE Conference on Visual Analytics Science and Technology (VAST’12). IEEE, 23–32.
[32]
Thomas S. Huang, Charlie K. Dagli, Shyamsundar Rajaram, Edward Y. Chang, Michael I. Mandel, Graham E. Poliner, and Daniel P. W. Ellis. 2008. Active Learning for Interactive Multimedia Retrieval. Proc. IEEE 96, 4 (Apr. 2008), 648– 667.
[33]
Weihua Huang and Chew Lim Tan. 2007. A system for understanding imaged infographics and its applications. In Proceedings of the ACM Symposium on Document Engineering (DocEng’07). ACM, 9–18.
[34]
Thorsten Joachims. 1999. Transductive inference for text classification using support vector machines. In Proceedings of the International Conference on Machine Learning (ICML’99). Morgan Kaufmann Publishers Inc., 200–209.
[35]
Bonnie Elizabeth John. 1988. Contributions to Engineering Models of Human-computer Interaction. Ph.D. Dissertation. Carnegie Mellon University.
[36]
Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. 2009. Multi-class active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). IEEE, 2372–2379.
[37]
Daekyoung Jung, Wonjae Kim, Hyunjoo Song, Jeong-in Hwang, Bongshin Lee, Bohyoung Kim, and Jinwook Seo. 2017. ChartSense: Interactive data extraction from chart images. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’17). ACM, 6706–6717.
[38]
Saurabh Kataria, William Browuer, Prasenjit Mitra, and Clyde Lee Giles. 2008. Automatic extraction of data points and text blocks from 2-dimensional plots in digital documents. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’08). AAAI Press, 1169–1174.
[39]
David E. Kieras. 1988. Towards a Practical GOMS model methodology for user interface design. In Handbook of Human-computer Interaction, Martin G. Helander (Ed.). Elsevier, 135–157.
[40]
Klokan Technologies GmbH. Old Maps Online. Retrieved on 31 May, 2020 from https://www.oldmapsonline.org/.
[41]
Kostiantyn Kucher, Carita Paradis, Magnus Sahlgren, and Andreas Kerren. 2017. Active learning and visual analytics for stance classification with ALVA. ACM Trans. Interact. Intell. Syst. 7, 3 (Oct. 2017).
[42]
Branislav Kveton and Shlomo Berkovsky. 2016. Minimal interaction content discovery in recommender systems. ACM Trans. Interact. Intell. Syst. 6, 2 (2016).
[43]
Christian Leistner, Amir Saffari, Jakob Santner, and Horst Bischof. 2009. Semi-supervised random forests. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). IEEE, 506–513.
[44]
David D. Lewis and Jason Catlett. 1994. Heterogeneous uncertainty sampling for supervised learning. In Proceedings of the International Conference on Machine Learning (ICML’94). Morgan Kaufmann Publishers Inc., 148–156.
[45]
Hongsen Liao, Li Chen, Yibo Song, and Hao Ming. 2016. Visualization-based active learning for video annotation. IEEE Trans. Multimedia 18, 11 (Nov. 2016), 2196–2205.
[46]
Library of Congress. Library of Congress Digital Collections. Retrieved on 31 May, 2020 from https://www.loc.gov/collections/.
[47]
Dong Liu, Meng Wang, Xian-Sheng Hua, and Hong-Jiang Zhang. 2009. Smart batch tagging of photo albums. In Proceedings of the ACM International Conference on Multimedia (MM’09). ACM, 809–812.
[48]
Ruizhe Liu, Weihua Huang, and Chew Lim Tan. 2007. Extraction of vectorized graphical information from scientific chart images. In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR’07). IEEE, 521–525.
[49]
Yiwen Luo, Wei Liu, Jianzhuang Liu, and Xiaoou Tang. 2008. MQSearch: Image search by multi-class query. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’08). ACM, 49–52.
[50]
Gonzalo Gabriel Méndez, Miguel A. Nacenta, and Sebastien Vandenheste. 2016. iVoLVER: Interactive visual language for visualization extraction and reconstruction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’16). ACM, 4073–4085.
[51]
Charles Joseph Minard. 1845. Carte de la Circulation des Voyageurs par Voitures Publiques sur les routes de la contrée où sera placé le Chemin de Fer de Dijon à Mulhouse. E. Thunot, Paris.
[52]
Charles Joseph Minard. 1859. Carte Figurative et approximative des tonnages des Grand Ports et des principales Rivières d’Europe. Règnier et Dourdet, Paris.
[53]
Charles Joseph Minard. 1869. Carte figurative des pertes successives en hommes de l’Armée Française dans la campagne de Russie 1812-1813. Règnier et Dourdet, Paris.
[54]
Hieu T. Nguyen and Arnold Smeulders. 2004. Active learning using pre-clustering. In Proceedings of the International Conference on Machine Learning (ICML’04). ACM, 79.
[55]
Florence Nightingale. 1858. Mortality of the British Army: At Home and Abroad, and during the Russian War, as Compared with the Mortality of the Civil Population in England. Harrison and Sons, London.
[56]
Florence Nightingale. 1858. Notes on Matters Affecting the Health, Efficiency, and Hospital Administration of the British Army. Harrison and Sons, London.
[57]
Alban William Housego Phillips. 1958. The relation between unemployment and the rate of change of money wage rates in the United Kingdom, 1861–1957. Economica 25, 100 (1958), 283–299.
[58]
William Playfair. 1786. The Commercial and Political Atlas: Representing, by Means of Stained Copper-Plate Charts, the Progress of the Commerce, Revenues, Expenditure, and Debts of England, during the Whole of the Eighteenth Century. Debrett, Robinson, and Sewell, London.
[59]
William Playfair. 1786. Exports & Imports to and from all North-America. Debrett, Robinson, and Sewell, London.
[60]
William Playfair. 1801. The Statistical Breviary: Shewing, on a Principle Entirely New, the Resources of Every State and Kingdom in Europe. Wallis, London.
[61]
William Playfair. 1821. A letter on our agricultural distresses, their causes and remedies: accompanied with tables and copper-plate charts, shewing and comparing the prices of wheat, bread and labour from 1565 to 1821. 1822. Digital Collections, NZ 822p, Yale University Library (Ed.). William Sams.
[62]
Jorge Poco and Jeffrey Heer. 2017. Reverse-engineering visualizations: Recovering visual encodings from chart images. Comput. Graph. For. 36, 3 (June 2017), 353–363.
[63]
Jorge Poco, Angela Mayhua, and Jeffrey Heer. 2018. Extracting and retargeting color mappings from bitmap images of visualizations. IEEE Trans. Vis. Comput. Graph. 24, 1 (Jan. 2018), 637–646.
[64]
Dimitrios Rafailidis, Apostolos Axenopoulos, Jonas Etzold, Stavroula Manolopoulou, and Petros Daras. 2014. Content-based tag propagation and tensor factorization for personalized item recommendation based on social tagging. ACM Trans. Interact. Intell. Syst. 3, 4 (Jan. 2014).
[65]
Ankit Rohatgi. 2011. WebPlotDigitizer. Retrieved on 31 May, 2020 from https://automeris.io/WebPlotDigitizer.
[66]
David Rumsey. David Rumsey Map Collection. Retrieved on 31 May, 2020 from https://www.davidrumsey.com/.
[67]
Bryan C. Russell, Antonio Torralba, Kevin P. Murphy, and William T. Freeman. 2008. LabelMe: A database and web-based tool for image annotation. Int. J. Comput. Vis. 77, 1-3 (2008), 157–173.
[68]
Dominik Sacha, Matthias Kraus, Daniel A. Keim, and Min Chen. 2019. VIS4ML: An ontology for visual analytics assisted machine learning. IEEE Trans. Vis. Comput. Graph. 25, 1 (Jan. 2019), 385–395.
[69]
Dominik Sacha, Michael Sedlmair, Leishi Zhang, John A. Lee, Jaakko Peltonen, Daniel Weiskopf, Stephen C. North, and Daniel A. Keim. 2017. What you see is what you can change: Human-centered machine learning by interactive visualization. Neurocomputing 268 (Dec. 2017), 164–175.
[70]
Manolis Savva, Nicholas Kong, Arti Chhajta, Li Fei-Fei, Maneesh Agrawala, and Jeffrey Heer. 2011. ReVision: Automated classification, analysis and redesign of chart images. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST’11). ACM, 393–402.
[71]
Christin Seifert and Michael Granitzer. 2010. User-based active learning. In Proceedings of the IEEE International Conference on Data Mining Workshops. IEEE, 418–425.
[72]
Burr Settles. 2009. Active Learning Literature Survey. Technical Report. University of Wisconsin–Madison.
[73]
Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Divvala, and Ali Farhadi. 2016. FigureSeer: Parsing result-figures in research papers. In Proceedings of the European Conference on Computer Vision (ECCV’16), Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, 664–680.
[74]
John Snow. 1854. Cholera Map. Charles Frederick Cheffins, London.
[75]
John Snow. 1855. On the Mode of Communication of Cholera. John Churchill, London.
[76]
Qixiang Tan (Ed.). 1982. The Historical Atlas of China. China Cartographic Publishing House.
[77]
Jinhui Tang, Qiang Chen, Meng Wang, Shuicheng Yan, Tat-Seng Chua, and Ramesh Jain. 2013. Towards optimizing human labeling for interactive image tagging. ACM Trans. Multimedia Comput., Commun., Applic. 9, 4 (Aug. 2013).
[78]
Yuandong Tian, Wei Liu, Rong Xiao, Fang Wen, and Xiaoou Tang. 2007. A face annotation framework with partial clustering and interactive labeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). IEEE, 1–8.
[79]
Wei Tong and Rong Jin. 2007. Semi-supervised learning by mixed label propagation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’07). AAAI Press, 651–656.
[80]
Bas Tummers. 2006. DataThief III. Retrieved on 31 May, 2020 from http://datathief.org/.
[81]
United States Census Bureau. United States Census Bureau Publications. Retrieved on 31 May, 2020 from https://www.census.gov/library/publications.html.
[82]
Jarke J. van Wijk. 2005. The value of visualization. In Proceedings of the IEEE Conference on Visualization (VIS’05). IEEE, 79–86.
[83]
Julie S. Weber and Martha E. Pollack. 2007. Entropy-driven online active learning for interactive calendar management. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’07). ACM, 141–150.
[84]
Zuobing Xu, Ram Akella, and Yi Zhang. 2007. Incorporating diversity and density in active learning for relevance feedback. In Proceedings of the European Conference on IR Research (ECIR’07). Springer-Verlag, 246–257.
[85]
David Yarowsky. 1995. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the Meeting of the Association for Computational Linguistics (ACL’95). Association for Computational Linguistics, 189–196.
[86]
Yu Zhang, Bob Coecke, and Min Chen. 2019. On the cost of interactions in interactive visual machine learning. In Proceedings of the IEEE VIS Workshop on Evaluation of Interactive Visual Machine Learning Systems. IEEE, 5.
[87]
Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, and Bernhard Schölkopf. 2003. Learning with Local and Global Consistency. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS’03). The MIT Press, 321–328.
[88]
Yan Ping Zhou and Chew Lim Tan. 2000. Hough Technique for Bar Charts Detection and Recognition in Document Images. In Proceedings of the International Conference on Image Processing (ICIP’00). IEEE, 605–608.
[89]
Xiaojin Zhu and Zoubin Ghahramani. 2002. Learning from Labeled and Unlabeled Data with Label Propagation. Technical Report. Carnegie Mellon University.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems
ACM Transactions on Interactive Intelligent Systems  Volume 11, Issue 3-4
December 2021
483 pages
ISSN:2160-6455
EISSN:2160-6463
DOI:10.1145/3481699
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2021
Accepted: 01 July 2021
Revised: 01 June 2020
Received: 01 November 2019
Published in TIIS Volume 11, Issue 3-4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Intelligent user interface
  2. active learning
  3. interactive classification
  4. data labeling
  5. interaction reduction
  6. data reconstruction
  7. historical visualization

Qualifiers

  • Research-article
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)35
  • Downloads (Last 6 weeks)7
Reflects downloads up to 03 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media