demonstration

Versatile Immersive Virtual and Augmented Tangible OR – Using VR, AR and Tangibles to Support Surgical Practice

Authors:

Anke Verena Reinschluessel,

Thomas Muender,

Roland Fischer,

Valentin Kraft,

Verena Nicole Uslar,

Gabriel Zachmann,

Rainer MalakaAuthors Info & Claims

CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 477, Pages 1 - 5

https://doi.org/10.1145/3544549.3583895

Published: 19 April 2023 Publication History

Abstract

Immersive technologies such as virtual reality (VR) and augmented reality (AR), in combination with advanced image segmentation and visualization, have considerable potential to improve and support a surgeon’s work. We demonstrate a solution to help surgeons plan and perform surgeries and educate future medical staff using VR, AR, and tangibles. A VR planning tool improves spatial understanding of an individual’s anatomy, a tangible organ model allows for intuitive interaction, and AR gives contactless access to medical images in the operating room. Additionally, we present improvements regarding point cloud representations to provide detailed visual information to a remote expert and about the remote expert. Therefore, we give an exemplary setup showing how recent interaction techniques and modalities benefit an area that can positively change the life of patients.

Supplementary Material

VTT File (3544549.3583895-walkthrough.vtt)

Download
1.71 KB

MP4 File (3544549.3583895-walkthrough.mp4)

Walkthrough Video

Download
104.84 MB

MP4 File (3544549.3583895-preview.mp4)

Video Preview

Download
25.12 MB

References

[1]

Freddie Bray, Jacques Ferlay, Isabelle Soerjomataram, Rebecca L Siegel, Lindsey A Torre, and Ahmedin Jemal. 2018. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a cancer journal for clinicians 68, 6 (2018), 394–424.

[2]

Roland Fischer, Andre Mühlenbrock, Farin Kulapichitr, Verena Nicole Uslar, Dirk Weyhe, and Gabriel Zachmann. 2022. Evaluation of Point Cloud Streaming and Rendering for VR-Based Telepresence in the OR. In Virtual Reality and Mixed Reality (EuroXR), Vol. 13484. Springer International Publishing, Cham, Switzerland, 89–110.

[3]

Danilo Gasques, Janet G. Johnson, Tommy Sharkey, Yuanyuan Feng, Ru Wang, Zhuoqun Robin Xu, Enrique Zavala, Yifei Zhang, Wanze Xie, Xinming Zhang, Konrad Davis, Michael Yip, and Nadir Weibel. 2021. ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical Telementoring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14.

Digital Library

[4]

Franklin King, Jagadeesan Jayender, Sharath K Bhagavatula, Paul B Shyn, Steve Pieper, Tina Kapur, Andras Lasso, and Gabor Fichtinger. 2016. An immersive virtual reality environment for diagnostic imaging. Journal of Medical Robotics Research 1, 01 (2016), 1640003.

[5]

Valentin Kraft, Florian Link, Andrea Schenk, and Christian Schumann. 2020. Adaptive illumination sampling for direct volume rendering. In Computer Graphics International Conference. Springer, Springer, Cham, Switzerland, 107–118.

Digital Library

[6]

Thomas Muender, Anke Verena Reinschluessel, Daniela Salzmann, Thomas Lück, Andrea Schenk, Dirk Weyhe, Tanja Döring, and Rainer Malaka. 2022. Evaluating Soft Organ-Shaped Tangibles for Medical Virtual Reality. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 237, 8 pages. https://doi.org/10.1145/3491101.3519715

Digital Library

[7]

Andre Mühlenbrock, Roland Fischer, Christoph Schröder-Dering, René Weller, and Gabriel Zachmann. 2022. Fast, accurate and robust registration of multiple depth sensors without need for RGB and IR images. The Visual Computer 38(2022), 1–14.

Digital Library

[8]

Pontus Olsson, Fredrik Nysjö, Jan-Michaél Hirsch, and Ingrid B. Carlbom. 2013. A haptics-assisted cranio-maxillofacial surgery planning system for restoring skeletal anatomy in complex trauma cases. International Journal of Computer Assisted Radiology and Surgery 8, 6 (Nov. 2013), 887–894.

[9]

Anke V. Reinschluessel, Roland Fischer, Christian Schumann, Verena Uslar, Thomas Muender, Uwe Katzky, Heike Kißner, Valentin Kraft, Marie Lampe, Thomas Lück, Klaus Bock-Müller, Hans Nopper, Sirko Pelzl, Dirk Wenig, Andrea Schenk, Dirk Weyhe, Gabriel Zachmann, and Rainer Malaka. 2019. Introducing Virtual & 3D-Printed Models for Improved Collaboration in Surgery. In Proceedings of the 18. Annual Meeting of the German Society of Computer- and Robot-Assisted Surgery, Oliver Burgert (Ed.). Vol. 18. Deutsche Gesellschaft für die Computer-und Roboterassistierte Chirurgie e.V., Stuttgart, Germany, 253–258.

[10]

Anke V. Reinschluessel, Thomas Muender, Tanja Döring, Verena Uslar, Thomas Lück, Dirk Weyhe, Andrea Schenk, and Rainer Malaka. 2021. A Study on the Size of Tangible Organ-Shaped Controllers for Exploring Medical Data in VR. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI ’21 Extended Abstracts). ACM, ACM, New York, NY, USA, 7.

Digital Library

[11]

Anke V Reinschluessel, Thomas Muender, Daniela Salzmann, Tanja Doering, Rainer Malaka, and Dirk Weyhe. 2022. Virtual Reality for Surgical Planning-Evaluation based on Two Liver Tumour Resections. Frontiers in Surgery 9(2022), 110.

[12]

Anke V. Reinschluessel, Thomas Muender, Verena Uslar, Dirk Weyhe, Andrea Schenk, and Rainer Malaka. 2019. Tangible Organs: Introducing 3D Printed Organ Models with VR to Interact with Medical 3D Models. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems(CHI EA ’19). ACM, New York, NY, USA, LBW1816:1–LBW1816:6.

Digital Library

[13]

Bernhard Reitinger, Alexander Bornik, Reinhard Beichel, and Dieter Schmalstieg. 2006. Liver Surgery Planning Using Virtual Reality. IEEE Computer Graphics and Applications 26, 6 (11 2006), 36–47.

Digital Library

[14]

Andrea Schenk, Dieter Haemmerich, and Tobias Preusser. 2011. Planning of image-guided interventions in the liver. IEEE pulse 2, 5 (2011), 48–55.

[15]

A. Schenk, S. Zidowitz, H. Bourquain, M. Hindennach, C. Hansen, H.K. Hahn, and H.-O. Peitgen. 2008. Clinical relevance of model based computer-assisted diagnosis and therapy. In Medical Imaging 2008: Computer-Aided Diagnosis, Vol. 6915. SPIE, Bellingham, WA, USA, 691502–1–691502–19.

[16]

Jeffrey H Shuhaiber. 2004. Augmented reality in surgery. Archives of surgery 139, 2 (2004), 170–174.

[17]

V Uslar, J Preibisch, N Tabriz, T Aumann-Münch, and D Weyhe. 2022. Nutzen der VR-Technik in der klinischen Lehre und der bildgebenden Diagnostik. Zeitschrift für Gastroenterologie 60, 08 (2022), KA481.

[18]

Junchen Wang, Hideyuki Suenaga, Kazuto Hoshi, Liangjing Yang, Etsuko Kobayashi, Ichiro Sakuma, and Hongen Liao. 2014. Augmented reality navigation with automatic marker-free image registration using 3-D image overlay for dental surgery. IEEE transactions on biomedical engineering 61, 4 (2014), 1295–1304.

[19]

Eiju Watanabe, Makoto Satoh, Takehiko Konno, Masahiro Hirai, and Takashi Yamaguchi. 2016. The trans-visible navigator: a see-through neuronavigation system using augmented reality. World neurosurgery 87(2016), 399–405.

[20]

Jeong Hee Yoon, Joon-Il Choi, Yong Yeon Jeong, Andrea Schenk, Longquan Chen, Hendrik Laue, So Yeon Kim, and Jeong Min Lee. 2016. Pre-treatment estimation of future remnant liver function using gadoxetic acid MRI in patients with HCC. Journal of hepatology 65, 6 (2016), 1155–1162.

Cited By

Weyhe DHartmann VUslar VTabriz N(2025)Augmented und Virtual Reality in der Chirurgie: Einsatzgebiete und explorative Studien am Beispiel von VIVATOPAugmented and virtual reality in surgery: fields of application and exploratory studies exemplified by VIVATOPDie Chirurgie10.1007/s00104-024-02218-3Online publication date: 3-Jan-2025
https://doi.org/10.1007/s00104-024-02218-3
Chiossi FEl Khaoudi YOu CSidenmark LZaky AFeuchtner TMayer S(2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698142
Tashiro YMiyafuji SKojima YKiyofuji SKin TIgarashi TKoike H(2024)MR Microsurgical Suture Training System with Level-Appropriate SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642324(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642324
Show More Cited By

Index Terms

Versatile Immersive Virtual and Augmented Tangible OR – Using VR, AR and Tangibles to Support Surgical Practice

Recommendations

Building and Employing Cross-Reality

This issue's Works in Progress department lists five projects with a focus on cross reality. The first two involve using Project Wonderland for collaborative projects. The next three use virtual and augmented reality to enhance user capabilities and ...
Journalism in virtual reality: opportunities and future research challenges
AcademicMindtrek '16: Proceedings of the 20th International Academic Mindtrek Conference

This paper presents a state-of-the-art overview of journalism and its opportunities and challenges in virtual reality (VR). First, we examine what kind of real-life journalistic experiments have been made in this field thus far, and then we analyze the ...
AR and VR Enhances Learning
Smart Learning for A Sustainable Society
Abstract
AR and VR technologies hold significant potential for enhancing learning by creating captivating and interactive environments. AR overlays digital information onto the real world, while VR generates fully simulated settings, offering realistic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

3914 pages

ISBN:9781450394222

DOI:10.1145/3544549

Editors:
Albrecht Schmidt
LMU Munich, Germany
,
Kaisa Väänänen
Tampere University, Finland
,
Tesh Goyal
Google Research, USA
,
Per Ola Kristensson
University of Cambridge, UK
,
Anicia Peters
University of Namibia, Namibia

Copyright © 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Check for updates

Badges

Honorable Mention

Author Tags

Qualifiers

Demonstration
Research
Refereed limited

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
503
Total Downloads

Downloads (Last 12 months)269
Downloads (Last 6 weeks)25

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Weyhe DHartmann VUslar VTabriz N(2025)Augmented und Virtual Reality in der Chirurgie: Einsatzgebiete und explorative Studien am Beispiel von VIVATOPAugmented and virtual reality in surgery: fields of application and exploratory studies exemplified by VIVATOPDie Chirurgie10.1007/s00104-024-02218-3Online publication date: 3-Jan-2025
https://doi.org/10.1007/s00104-024-02218-3
Chiossi FEl Khaoudi YOu CSidenmark LZaky AFeuchtner TMayer S(2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698142
Tashiro YMiyafuji SKojima YKiyofuji SKin TIgarashi TKoike H(2024)MR Microsurgical Suture Training System with Level-Appropriate SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642324(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642324
Schenk AKluge APelzl SZachmann GMalaka R(2024)Von der Bildgebung zur Interaktion mit 3D-Modellen: technische AspekteFrom imaging to interaction with 3D models: technical aspectsDie Chirurgie10.1007/s00104-024-02214-7Online publication date: 23-Dec-2024
https://doi.org/10.1007/s00104-024-02214-7
Kleinbeck CSmietana MLewis NTeufel TKreimeier JHeinzl CSteiner JAnthes CRoth D(2023)Adaptive Volumetric Anatomy Visualization in VR with Tangible Control2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00131(613-614)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00131
Reinschluessel AZagermann J(2023)Exploring Hybrid User Interfaces for Surgery Planning2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00048(208-210)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR-Adjunct60411.2023.00048

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Table of Contents