research-article

Projection Path Explorer: Exploring Visual Patterns in Projected Decision-making Paths

Authors:

Andreas Hinterreiter,

Christian Steinparz,

Moritz SchÖfl,

Marc StreitAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 11, Issue 3-4

Article No.: 22, Pages 1 - 29

https://doi.org/10.1145/3387165

Published: 03 September 2021 Publication History

Abstract

In problem-solving, a path towards a solutions can be viewed as a sequence of decisions. The decisions, made by humans or computers, describe a trajectory through a high-dimensional representation space of the problem. By means of dimensionality reduction, these trajectories can be visualized in lower-dimensional space. Such embedded trajectories have previously been applied to a wide variety of data, but analysis has focused almost exclusively on the self-similarity of single trajectories. In contrast, we describe patterns emerging from drawing many trajectories—for different initial conditions, end states, and solution strategies—in the same embedding space. We argue that general statements about the problem-solving tasks and solving strategies can be made by interpreting these patterns. We explore and characterize such patterns in trajectories resulting from human and machine-made decisions in a variety of application domains: logic puzzles (Rubik’s cube), strategy games (chess), and optimization problems (neural network training). We also discuss the importance of suitably chosen representation spaces and similarity metrics for the embedding.

Supplementary Material

hinterreiter (hinterreiter.zip)

Supplemental movie, appendix, image and software files for, Projection Path Explorer: Exploring Visual Patterns in Projected Decision-making Paths

Download
385.49 KB

References

[1]

Natalia Andrienko and Gennady Andrienko. 2013. Visual analytics of movement: An overview of methods, tools and procedures. Inf. Vis. 12, 1 (2013), 3–24.

Digital Library

[2]

Mathieu Aubry and Bryan C. Russell. 2015. Understanding deep features with computer-generated imagery. In IEEE International Conference on Computer Vision (ICCV’15). IEEE, 2875–2883.

[3]

Benjamin Bach, Conglei Shi, Nicolas Heulot, Tara Madhyastha, Tom Grabowski, and Pierre Dragicevic. 2016. Time curves: Folding time to visualize patterns of temporal evolution in data. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 559–568. Retrieved from http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=7192639.

Digital Library

[4]

Jürgen Bernard, Nils Wilhelm, and Maximilian Scherer. 2012. TimeSeriesPaths: Projection-based explorative analysis of multivariate time series data. J. WSCG 20, 2 (2012), 97–106.

[5]

Hasan Alp Boz. 2019. A Visual Multivariate Dynamic EgoCentric Network Exploration Tool. Master’s thesis. Sabanci University, Istanbul, Turkey.

[6]

Eli T. Brown, Sriram Yarlagadda, Kristin A. Cook, Alex Endert, and Remco Chang. 2018. ModelSpace: Visualizing the trails of data models in visual analytics systems. In MLUI 2018: Machine Learning from User Interactions for Visualization and Analytics. IEEE, Berlin, Germany.

[7]

Eren Cakmak, Daniel Seebacher, Juri Buchmuller, and Daniel A. Keim. 2018. Time series projection to highlight trends and outliers. In IEEE Conference on Visual Analytics Science and Technology (VAST’18). IEEE, Berlin, Germany, 104–105.

[8]

Ricardo J. G. B. Campello, Davoud Moulavi, and Joerg Sander. 2013. Density-based clustering based on hierarchical density estimates. In Advances in Knowledge Discovery and Data Mining, David Hutchison, Takeo Kanade, Josef Kittler, Jon M. Kleinberg, Friedemann Mattern, John C. Mitchell, Moni Naor, Oscar Nierstrasz, C. Pandu Rangan, Bernhard Steffen, Madhu Sudan, Demetri Terzopoulos, Doug Tygar, Moshe Y. Vardi, Gerhard Weikum, Jian Pei, Vincent S. Tseng, Longbing Cao, Hiroshi Motoda, and Guandong Xu (Eds.). Vol. 7819. Springer Berlin, 160–172.

[9]

S. Chinchalkar. 1996. An upper bound for the number of reachable positions. ICGA J. 19, 3 (1996), 181–183.

[10]

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. DeCAF: A deep convolutional activation feature for generic visual recognition. In Proceedings of Machine Learning Research, Vol. 32. PMLR, 647–655. Retrieved from http://arxiv.org/abs/1310.1531.

[11]

Fan Du, Nan Cao, Jian Zhao, and Yu-Ru Lin. 2015. Trajectory bundling for animated transitions. In 33rd ACM Conference on Human Factors in Computing Systems (CHI’15). ACM, New York, NY, 289–298.

Digital Library

[12]

Daniel Engel, Lars Hüttenberger, and Bernd Hamann. 2012. A survey of dimension reduction methods for high-dimensional data analysis and visualization. In Visualization of Large and Unstructured Data Sets: Applications in Geospatial Planning, Modeling and Engineering – Proceedings of IRTG 1131 Workshop 2011 (OpenAccess Series in Informatics (OASIcs)), Vol. 27. Schloss Dagstuhl–Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 135–149.

[13]

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why does unsupervised pre-training help deep learning?J. Mach. Learn. Res. 11 (2010), 625–660.

Digital Library

[14]

O. Ersoy, C. Hurter, F. V. Paulovich, G. Cantareiro, and A. Telea. 2011. Skeleton-based edge bundling for graph visualization. IEEE Trans. Vis. Comput. Graph. 17, 12 (2011), 2364 –2373.

Digital Library

[15]

Mateus Espadoto, Rafael M. Martins, Andreas Kerren, Nina S. T. Hirata, and Alexandru Cristian Telea. 2019. Towards a quantitative survey of dimension reduction techniques. IEEE Trans. Vis. Comput. Graph. (2019).

[16]

Jessica Fridrich. 1997. My system for solving Rubik’s cube. (1997). Retrieved from http://www.ws.binghamton.edu/fridrich/system.html.

[17]

Frederic Friedel. 2017. Fritz 16 – Freund und Trainer. ChessBase. (2017). Retrieved from https://de.chessbase.com/post/fritz-16-freund-und-trainer.

[18]

Anna Gogolou, Theophanis Tsandilas, Themis Palpanas, and Anastasia Bezerianos. 2019. Comparing similarity perception in time series visualizations. IEEE Trans. Vis. Comput. Graph. 25, 1 (2019), 523–533.

Digital Library

[19]

Philippe Hamel and Douglas Eck. 2010. Learning features from music audio with deep belief networks. In 11th International Society for Music Information Retrieval Conference. International Society for Music Information Retrieval, Utrecht, Netherlands, 339–344.

[20]

Pierre Havard. 2019. KingBase – a free chess games database, updated monthly. Retrieved from https://www.kingbase-chess.net/.

[21]

Jiangen He and Chaomei Chen. 2017. Visualizing temporal patterns in representation data. In VADL 2017: Workshop on Visual Analytics for Deep Learning. IEEE.

[22]

Andreas Hinterreiter, Peter Ruch, Holger Stitz, Martin Ennemoser, Jürgen Bernard, Hendrik Strobelt, and Marc Streit. 2020. ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion. Retrieved from http://arxiv.org/abs/1910.00969.

[23]

G. E. Hinton. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504–507.

[24]

Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Trans. Vis. Comput. Graph. 25, 8 (2018), 2674–2693.

Digital Library

[25]

Danny Holten and Jarke J. van Wijk. 2009. Force-directed edge bundling for graph visualization. Comput Graph Forum (EuroVis’09) 28, 3 (2009), 983–990.

[26]

Christophe Hurter, Ozan Ersoy, Sara I. Fabrikant, Tijmen R. Klein, and Alexandru C. Telea. 2013. Bundled visualization of dynamic graph and trail data. IEEE Trans. Vis. Comput. Graph. 20, 8 (2013), 1141–1157.

Digital Library

[27]

Brittany Kondo and Christopher Collins. 2014. DimpVis: Exploring time-varying information visualizations by direct manipulation. IEEE Trans. Vis. Comput. Graph. (InfoVis’14) 20, 12 (2014), 2003–2012.

[28]

Yann LeCun. 2005. The MNIST database of handwritten digits. Retrieved from http://yann.lecun.com/exdb/mnist/.

[29]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436–444.

[30]

Fritz Lekschas, Michael Behrisch, Benjamin Bach, Peter Kerpedjiev, Nils Gehlenborg, and Hanspeter Pfister. 2019. Pattern-driven navigation in 2D multiscale visualizations with scalable insets.

[31]

Lucas Liberacki and Tom Brannan. 2015. Rubik’s Cube Solver coded in Python. Retrieved from https://github.com/CubeLuke/Rubiks-Cube-Solver.

[32]

Wei-Li Lu, Yu-Shuen Wang, and Wen-Chieh Lin. 2014. Chess evolution visualization. IEEE Trans. Vis. Comput. Graph. 20, 5 (2014), 702–713.

Digital Library

[33]

Yi Mao, Joshua Dillon, and Guy Lebanon. 2007. Sequential document visualization. IEEE Trans. Vis. Comput. Graph. 13, 6 (2007), 1208–1215.

Digital Library

[34]

Leland McInnes, John Healy, and James Melville. 2018. UMAP: Uniform manifold approximation and projection for dimension reduction. Retrieved from http://arxiv.org/abs/1802.03426.

[35]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533.

[36]

Abdel-rahman Mohamed, Geoffrey Hinton, and Gerald Penn. 2012. Understanding how deep belief networks perform acoustic modelling. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’12). IEEE, 4273–4276.

[37]

Nikolay Oskolkov. 2019. How Exactly UMAP Works. Retrieved from https://towardsdatascience.com/how-exactly-umap-works-13e3040e1668.

[38]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in python. J. Mach. Learn. Res. 12 (2011), 2825–2830.

Digital Library

[39]

Lars Petrus. 1997. Solving Rubik’s Cube for speed.Retrieved from https://lar5.com/cube/:2019-11-20.

[40]

V. Peysakhovich, C. Hurter, and A. Telea. 2015. Attribute-driven edge bundling for general graphs with applications in trail analysis. In IEEE Pacific Visualization Symposium (PacificVis’15). IEEE, 39–46.

[41]

Pavlin G. Poličar, Martin Stražar, and Blaž Zupan. 2019. openTSNE: A modular python library for t-SNE dimensionality reduction and embedding.

[42]

P. E. Rauber, S. G. Fadel, A. X. Falcão, and A. C. Telea. 2017. Visualizing the hidden activity of artificial neural networks. IEEE Trans. Vis. Comput. Graph. 23, 1 (2017), 101–110.

Digital Library

[43]

Tomas Rokicki, Herbert Kociemba, Morley Davidson, and John Dethridge. 2010. God’s Number is 20. Retrieved from http://www.cube20.org/.

[44]

Tobias Schreck, Tatiana Tekušová, Jörn Kohlhammer, and Dieter Fellner. 2007. Trajectory-based visual analysis of large financial time series data. ACM SIGKDD Explor. Newslett. 9, 2 (2007), 30–37.

Digital Library

[45]

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140–1144.

[46]

Christian Steinparz, Andreas Hinterreiter, Holger Stitz, and Marc Streit. 2019. Visualization of Rubik’s cube solution algorithms. In EuroVis Workshop on Visual Analytics (EuroVA’19). The Eurographics Association.

[47]

Holger Stitz, Samuel Gratzl, Wolfgang Aigner, and Marc Streit. 2016. ThermalPlot: Visualizing multi-attribute time-series data using a thermal metaphor. IEEE Trans. Vis. Comput. Graph. 22, 12 (2016), 2594–2607.

Digital Library

[48]

J. B. Tenenbaum. 2000. A global geometric framework for nonlinear dimensionality reduction. Science 290, 5500 (2000), 2319–2323.

[49]

Deniz (https://github.com/dn1z). 2018. pgn2gif. Retrieved from https://github.com/dn1z/pgn2gif.

[50]

Henry (https://math.stackexchange.com/users/6460/henry). 2017. How is the distance of two random points in a unit hypercube distributed?Mathematics Stack Exchange. Retrieved from https://math.stackexchange.com/q/1985698.

[51]

S. van den Elzen, D. Holten, J. Blaas, and J. J. van Wijk. 2016. Reducing snapshots to points: A visual analytics approach to dynamic network exploration. IEEE Trans. Vis. Comput. Graph. 22, 1 (2016), 1–10.

Digital Library

[52]

Laurens van der Maaten. 2020. t-SNE – FAQ. (2020). Retrieved from https://lvdmaaten.github.io/tsne/#faq.

[53]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, Nov. (2008), 2579–2605.

[54]

Laurens van der Maaten, Eric Postma, and Jaap van den Herik. 2009. Dimensionality Reduction: A Comparative Review. Technical Report TR 2009-005. Tilburg University, Tilburg, Netherlands.

[55]

Zhiguang Wang and Tim Oates. 2015. Encoding time series as images for visual inspection and classification using tiled convolutional neural networks. In Trajectory-based Behavior Analytics: Papers from the 2015 AAAI Workshop. AAAI Press, 40–46.

[56]

Matthew O. Ward and Zhenyu Guo. 2011. Visual exploration of time-series data with shape space projections. Comput. Graph. Forum 30, 3 (2011), 701–710.

Digital Library

[57]

Yu Zheng. 2015. Trajectory data mining: An overview. ACM Trans. Intell. Syst. Technol. 6, 3 (2015), 1–41.

Digital Library

[58]

Biao Zhu and Wei Chen. 2016. Performance histogram curve: Abstractive visualization and analysis of NBA games. Int. J. Softw. Inform. 10, 3 (2016).

Cited By

Bittner MHinterreiter AEckelt KStreit M(2025)Explainable Long and Short-Term Pattern Detection in Projected Sequential DataMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74633-8_4(53-68)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74633-8_4
Humer CNicholls RHeberle HHeckmann MPühringer MWolf TLübbesmeyer MHeinrich JHillenbrand JVolpin GStreit M(2024)CIME4R: Exploring iterative, AI-guided chemical reaction optimization campaigns in their parameter spaceJournal of Cheminformatics10.1186/s13321-024-00840-116:1Online publication date: 10-May-2024
https://doi.org/10.1186/s13321-024-00840-1
Wallner GDrachen A(2024)Visualization of Player Movement Patterns with Line Integral Convolution and Alpha ShapesProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3649997(1-10)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3649997
Show More Cited By

Index Terms

Projection Path Explorer: Exploring Visual Patterns in Projected Decision-making Paths
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation types and techniques
      1. Visual analytics
2. Human-centered computing
  1. Visualization

Recommendations

Lorentzian discriminant projection and its applications
ACCV'09: Proceedings of the 9th Asian conference on Computer Vision - Volume Part III

This paper develops a supervised dimensionality reduction method, Lorentzian Discriminant Projection (LDP), for discriminant analysis and classification Our method represents the structures of sample data by a manifold, which is furnished with a ...
Locality-Preserved Maximum Information Projection

Dimensionality reduction is usually involved in the domains of artificial intelligence and machine learning. Linear projection of features is of particular interest for dimensionality reduction since it is simple to calculate and analytically analyze. ...
Supervised data-dependent kernel sparsity preserving projection for image recognition

Dimensionality reduction methods have commonly been used as principled ways to understand high-dimensional data. In this paper, a novel non-linear method named supervised data-dependent kernel sparsity preserving projection (SDKSPP) is proposed for ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 11, Issue 3-4

December 2021

483 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3481699

Editor:
Michelle X. Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2021

Accepted: 01 July 2020

Revised: 01 April 2020

Received: 01 November 2019

Published in TIIS Volume 11, Issue 3-4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

State of Upper Austria and the Austrian Federal Ministry of Education, Science and Research via the LIT–Linz Institute of Technology
State of Upper Austria (Human-Interpretable Machine Learning)
Austrian Research Promotion Agency
Austrian Science Fund

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
297
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)4

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bittner MHinterreiter AEckelt KStreit M(2025)Explainable Long and Short-Term Pattern Detection in Projected Sequential DataMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74633-8_4(53-68)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74633-8_4
Humer CNicholls RHeberle HHeckmann MPühringer MWolf TLübbesmeyer MHeinrich JHillenbrand JVolpin GStreit M(2024)CIME4R: Exploring iterative, AI-guided chemical reaction optimization campaigns in their parameter spaceJournal of Cheminformatics10.1186/s13321-024-00840-116:1Online publication date: 10-May-2024
https://doi.org/10.1186/s13321-024-00840-1
Wallner GDrachen A(2024)Visualization of Player Movement Patterns with Line Integral Convolution and Alpha ShapesProceedings of the 19th International Conference on the Foundations of Digital Games10.1145/3649921.3649997(1-10)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3649921.3649997
Dennig FMiller MKeim DEl-Assady M(2024)FS/DS: A Theoretical Framework for the Dual Analysis of Feature Space and Data SpaceIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328835630:8(5165-5182)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3288356
Takagi YTabalba RKirshenbaum NLeigh J(2024)Abstracted Trajectory Visualization for Explainability in Reinforcement Learning2024 IEEE Conference on Artificial Intelligence (CAI)10.1109/CAI59869.2024.00023(75-82)Online publication date: 25-Jun-2024
https://doi.org/10.1109/CAI59869.2024.00023
Rafieian BHermosilla PVázquez P(2023)Improving Dimensionality Reduction Projections for Data VisualizationApplied Sciences10.3390/app1317996713:17(9967)Online publication date: 4-Sep-2023
https://doi.org/10.3390/app13179967
Metz YBykovets EJoos LKeim DEl‐Assady M(2023)VISITOR: Visual Interactive State Sequence Exploration for Reinforcement LearningComputer Graphics Forum10.1111/cgf.1483942:3(397-408)Online publication date: 27-Jun-2023
https://doi.org/10.1111/cgf.14839
Scimone AEckelt KStreit MHinterreiter A(2023)Marjorie: Visualizing Type 1 Diabetes Data to Support Pattern ExplorationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332693630:1(1216-1226)Online publication date: 24-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3326936
Eckelt KHinterreiter AAdelberger PWalchshofer CDhanoa VHumer CHeckmann MSteinparz CStreit M(2023)Visual Exploration of Relationships and Structure in Low-Dimensional EmbeddingsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.315676029:7(3312-3326)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3156760
Walchshofer CHinterreiter AXu KStitz HStreit M(2023)Provectories: Embedding-Based Analysis of Interaction Provenance DataIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.313569729:12(4816-4831)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1109/TVCG.2021.3135697
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents