This paper explores the idealised ancient China envisioned by 17th-century Europeans through surrealist interactive narratives within virtual spaces. The VR interactive game Moonlit Windows immerses players in a narrative environment featuring integrated physical interaction. Players control the moon via an interactive orrery, travelling between different windows and experiencing shifts in time and space. Through literature review and historical research, the theoretical analysis of surreal aesthetics describes the process of constructing a virtual space, including context-specific worldviews, interaction mechanisms, and artistic elements. Feedback from participants collected during the exhibition led to iterative improvements in interaction design and sensory immersion and contributed to the discourse on interactive narrative design in VR.
1 Introduction
This project presents a utopian VR fantasy of China, depicting the experiences of a 17th-century Italian Jesuit traveling through China in his dreams. Players will immerse themselves in an art style that subverts reality, utilising an Orrery, a physical Earth-Moon Rotation device that serves as the key to exploring the contents of this diary.
In the game, players have the capability to interact with the Orrery, which is a physical Earth-Moon Rotation System device, enabling them to adjust the position and phase of the moon within the virtual reality environment. Through the movement of the moon, players’ visualisation of the external scenery outside the window will be heightened. Enigmatic windows will materialize as the moon traverses specific areas, providing panoramic views that encompass the player’s surroundings. By controlling the moon’s journey between these windows, players can prompt scene transitions. They will discover clues within the depicted scenes, gradually unveiling the missionary’s narrative in the East.
2 The 17th to 18th Century European Dream of China
In the 17th and 18th centuries, Europe was enthralled by a fervent fascination with the East, an intense admiration for the mystery and allure of the Far East that permeated European society. Eastern regions’ cultures, arts, and philosophies, such as China, India and Persia[Lach 1991], began to evoke widespread interest and imitation in Europe.
As information from the East increasingly reached the West, many European scholars, often without ever visiting the East, began researching it, primarily focusing on their idealised vision of China.
Athanasius Kircher, a renowned Jesuit scholar, scientist, and archaeologist of the 17th century, is regarded as one of the foremost China experts of his era due to his extensive study of China [D’Orban 1989]and his deep fascination with the Chinese language and culture. Although he never personally visited China, his book China Illustrata [Freddolini 2020]gained popularity and became one of the most important sources of information about China for Europeans at the time. The book can be seen as a whimsical encyclopedia of the Chinese empire, featuring numerous maps and illustrations that cover various topics, including Chinese landscapes, cities, architecture, flora and fauna, legend[Szczesniak 1952]. Accounts included descriptions of land vehicles for navigation, birds trained to aid fishermen in catching fish, alluring beauties, flying turtles, and fruits of immense proportions. He also reflected the perspective of the Jesuits in their missionary work in China in his book, emphasising the Christian elements of Chinese history.
Figure 1:
Figure 1: Detail of "sailing coaches", China Illustrata, 1671.
While China Illustrata garnered significant attention in its time, it is essential to note that Kircher’s descriptions and illustrations contain several misrepresentations and inaccuracies, as he never conducted research in China.
In the 17th and 18th centuries, European society was steeped in a profound sense of exoticism when it came to China. China was viewed as a mysterious, distant, and enigmatic realm, and this exotic fascination resonated with the Surrealist pursuit of the irrational and the unique. As a utopian concept, China played a significant role in shaping Italian and European intellectual history. For some proponents of the Enlightenment, China served as a vibrant metaphor for the potential new forms of governance and society. It symbolised an idealised ’other civilisation’.
It is evident that this idealised perception of China compelled Europe to reevaluate itself in the presence of another ancient and equally significant civilisation, leading to the discovery of the relativity of values and the importance of cosmopolitanism[Santangelo 2007]. It was more than mere exoticism; it served as a means to escape from reality or as a refuge for fantasies. Instead, it became the focal point of critical reflection during that era [Said 2007].
Surrealism emphasised an anti-conventional and irrational way of thinking[Nadeau 1965], and the Chinese craze transcended the boundaries of traditional European art and culture during this period. Artists and writers began adopting unconventional techniques to express their whimsical ideas, aligning with the spirit of Surrealism. The culture and art of this era challenged traditional boundaries, emphasised exotic and fantastic elements, and pursued the irrational and the bizarre, all of which were core concepts advocated by Surrealist aesthetics.
3 Narrative Forms of Surrealist Art
Drawing on the theory of surreal aesthetics, this project shows visual and interactive forms of 17th-century European perceptions of China. The narrative form of Surrealist art originates from a belief in the omnipotence of dreams and an understanding of the power of the unconscious mind[Cramer and Grant 2020]. Breton underscored the significance of exploring the subconscious and the world of dreams, advocating for a sense of reality that transcended rational thought[Bohn 1977] This stance expressed a rebellious attitude towards reason, tradition, and societal norms. Suggestive interactive elements in VR enable players to explore and strengthen their subconscious thoughts and experiences. Inspired by Sigmund Freud’s ideas, Surrealists regarded dreams as visual manifestations of hidden thoughts and desires [Jones 1910]. By actively engaging with these concealed visual elements, players can develop a deeper understanding of the virtual environment’s significance through imaginative exploration.
At the same time, Surrealists were known for their rejection of rigid definitions and constraints of art. Max Ernst was known for his expertise in blending various art forms to disrupt rational image interpretations. His painting Two Children Threatened by a Nightingale [of Modern Art 2017] incorporates drawing, writing, collage, and relief techniques. The artwork presents a deep perspective space with a striking linear composition, naturally allowing the classical architecture in the distance to merge with the solid woodblock structures in the foreground. Doors and handles positioned at the edges of the frame invite the player to enter the world depicted within the painting. Max Ernst was known for his expertise in blending various art forms to disrupt rational image interpretations. This approach is a testament to the potential of pioneering innovative artistic techniques in creating immersive environments.
4 Constructing Narratives and Interactions in Multiple Contexts
The project’s conceptual framework is rooted in the exploration of 17th-century European fantasies of China, conveyed through surrealistic storytelling.
The player is immersed in a dream world experienced by a European Jesuit missionary, gaining insight into how Europeans in the 17th century imagined and perceived China. This experience reflects a Western interpretation and redefinition of Chinese culture, shaped by its distinct historical and cultural context. This cross-cultural perspective is inspired by Fragment, a poem by Chinese poet Bian Zhilin[Barnstone and Ping 2010].
You are standing on a bridge enjoying the view, Someone’s watching you from a balcony. The moon adorns your window, You adorn someone else’s dream.
Guided by this poem and drawing upon the multiple associations in Surrealist art, the present study explores the intricate relationships and interconnections between individuals and environments[Baghramian and Carter 2019]. The worldview depicted in Moonlit Windows is metaphorically rich, with two central elements: the ’bright moon’ and the ’window.’ By skillfully enhancing the positive interplay among these suggestive elements, audiences are encouraged to engage their imagination in interpreting the connotations and meanings of the virtual environment[Cramer and Grant 2020].The bright moon serves as a constant companion adorning the Jesuit missionary’s window throughout his extensive journey from Europe to China. Meanwhile, symbolising his limited scope of understanding due to cultural differences rather than a complete grasp of truth, the critical element of windows represents his travels through various settings such as ship cabins, carriages, or rooms while attempting to learn about Chinese culture[Szczesniak 1952].
4.1 Virtual Reality Environments and Narratives
Moonlit Windows offers players an immersive watercolour-style visual experience, utilising stylised visuals to transport them into a fantastical period of East-West exchanges. The virtual environment initially presents the beginning of a 17th-century European missionary’s journey, set in a Jesuit’s studying room. Players are encouraged to explore the room to gain an initial understanding of the missionary’s personal traits and historical context. In the monthly weather shorts experienced by the Italian Jesuit, players were immersed in the diverse landscapes he observed through the moonlit windows.
Meanwhile, the moonlight also illuminates the eastern lands. This project presents players with surrealist art-infused scenes of Qing Dynasty Guangzhou ports and towns, featuring prominent locations such as wharves, streets, shops, temples, and ship cabins. Players can explore hidden tea puzzles within the bustling ancient Chinese trade ports in these diverse settings.
The virtual world was built in Unity, utilising a combination of 3D models, watercolor shaders, and 2D decals to craft the environment. The composition and layout of the scenes draw on surrealist art, creating an immersive 3D collage of Chinese elements that aligns with the project’s conceptual framework.
4.2 Virtual and Physical Interaction
Within its conceptual framework, Moonlit Windows revolves around the interplay between the moon and windows, propelling the narrative through the collection of diverse letters and interaction with the environment[Tyler 2023]. This project encompasses two fundamental gameplay mechanics: control the moon rotation to illuminate windows and collect letters (of the alphabet) to change the environment.
Through integrate diverse art forms and subvert rational interpretations of images, the project has developed a physically interactive installation depicting the Earth-Moon rotation, symbolising the product of the European Age of Sail. The design aims to enhance immersion by allowing players to experience changes in place and time within the narrative by manipulating the moon’s position and phase using a rotating handle on the Earth-Moon Rotation Device. As the moon passes through a window during this process, it projects light that simulates the Tyndall effect[Helmenstine 2020]. This visual feedback activates scenes outside the window, enabling players to traverse different environments. The project uses hand-tracking in VR to offer players a controller-free interaction experience, allowing direct manipulation of objects in virtual and physical spaces. It means players can touch and grab objects in the virtual space with their hands and alter the virtual environment by controlling a physical Orrery. The integration of virtual reality with physical devices aims to provide players with a visual narrative and interactive experience that closely combines surrealist art with specific historical contexts within multiple settings.
Figure 2:
Figure 2: Interactive Orrery and hand-tracking, London, 2023.
In July 2023, the V1 combined physical devices with virtual reality to interact. This Version elucidates the fundamental interaction between the moon and the windows in constructing a worldview. The physical devices were conducted to control a virtual environment within Unity, where the moon dynamically interacts with the window. Through technical testing, the project successfully demonstrated initial effective interaction and a virtual space in Unity3D. As the moon passes through a window, it emits light while smoothly transitioning to another 3D space. The efficiency of the connection between Arduino and Unity was verified, enabling dynamic movement of the moon in the virtual environment and real-time changes in its phases.
Figure 4:
Figure 4: Interaction tests V1 and V2, London, 2023.
Version 2
Moonlit Windows V2 successfully developed its initial demo, showcasing a comprehensive user flow that introduces interactions and guides players through various scenarios. The preliminary prototype of the Earth-Moon Rotation Box, an interactive physical device, has been completed to enable real-time interaction with the virtual reality environment. However, the interactive device of the project exhibits structural deficiencies and usability issues that necessitate improvement and updating. Additionally, conflicts arose between the data transfer ports of the interactive orrery and the Oculus Quest 2 headset.
5.2 Ars Electronica 2023
From September 6 to 10, 2023, Moonlit Windows, a VR game inspired by 17th-Century European Fascination with China, showed at the Ars Electronica in Linz, Austria. Moonlit Windows V3 utilises an ESP32-S2 microcontroller and the VR headset for data transmission via WiFi, enabling an effective connection between the interactive Orrery and the Oculus Quest 2.
During the exhibition, the project received valuable feedback from audiences. The game’s thematic elements and artistic effects garnered positive reviews, while constructive suggestions regarding VR interaction were made. Based on feedback from 89% of the audience, the captivating theme revolving around 17th-century European Chinese fantasies, focusing on the adventures of Italian Jesuit missionaries in China, received high praise due to its artistically stylised 3D art. The exquisite watercolour-like conveyed a sense of historical context and deeply immersed the audience in the virtual world.
In terms of VR interaction, it was deemed engaging but still required improvements in user experience. Moonlit Windows offered a combination of entertaining physical and virtual interactions. However, there were instances where guidance for interaction could have been clearer upon entering the VR space, leading to confusion among most audiences. Those feedbacks highlights the necessity to enhance user guidance, ensuring audiences can easily comprehend how to interact with the game. The other aspect raised in feedback concerns responsiveness within the virtual environment. Audiences found that interactions lacked sensitivity, requiring enhancements to reflect their actions in-game, which may necessitate technical improvements accurately.
Figure 5:
Figure 5: Exhibited at the Ars Electronica, Austria, 2023.
6 Conclusion
This paper elucidates the process of constructing interactive narratives for VR games within various contexts, necessitating the establishment of a coherent worldview and an audience-centric interactive narrative. The development of this worldview relies on extensive research encompassing diverse backgrounds, theories, and reference works from multiple contexts to foster a range of narrative forms. Simultaneously, establishing connections across diverse contexts serves to augment the logical cohesiveness of the narrative.
The interaction mechanics can be grounded in establishing a comprehensive worldview, while interactive elements can be tailored according to the thematic framework. In this context, Interaction design necessitates a focus on audience-centricity to accomplish the objective of an immersive experience. The primary interaction mechanics should consider the audience’s role in the narrative and their pursued objectives. It is crucial for the audience to perceive themselves as pivotal players in the story, with their choices and actions directly influencing its progression. Additionally, conducting thorough testing and gathering feedback is essential for developing VR interactions. Ensuring that the audience finds satisfaction in interacting with the game’s narrative and continuously improving it to meet their needs is imperative.
The surrealist aesthetic claims and representative works offer theoretical support and artistic references for developing virtual reality spaces, aligning with the concepts of automatism and subversion. By incorporating elements that encourage suggestive interactions within virtual reality environments, viewers are able to actively explore and engage with these elements, resulting in the creation of unconscious compositions through interconnecting simple, abstract visual elements that promote multiple associations. Additionally, Surrealism’s emphasis on placing subject matter at the core of artistic style and technique, integrating various art forms to serve narrative purposes, and prioritising effectiveness in communicating themes and ideas provides valuable insights for crafting interactive narratives in virtual reality.
Acknowledgments
Acknowledgements go to my good partner Jing Li (designer and programmer). Heartfelt thanks to my tutors: Zisch Fiona, Stephen Gage, Zisch Dominik, and my family: Huibo Hu and my kitty.
References
[1]
Maria Baghramian and J. Adam Carter. 2019. Relativism. Routledge.
Charles D’Orban. 1989. Lexicography in China: bibliography of dictionaries and related literature. Bulletin of the School of Oriental and African Studies 52, 1 (1989), 207–208.
Francesco Freddolini. 2020. (Re) imagining Asian Rulers in Athanasius Kircher’s China Illustrata: The Agency of Interiors. RACAR: Revue d’art canadienne 45, 2 (2020), 64–80.
TEI '18: Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction
We present two proof of concept sensory experiences designed for virtual reality (VR). Our experiences bring together smell, sound, taste, touch, and sight, focusing on low-cost, non-digital materials and on passive interactions. We also contribute a ...
Design, User Experience, and Usability: Theory and Practice
Abstract
When the immersive filmwork is experienced by the audiences with head-mounted display,the interactive experience of immersive images lies in that the audiences (users) need to be part of the story in terms of the visual perception experience and ...
In this paper, we present a new immersive multiplayer game system developed for two different environments, namely, virtual reality (VR) and augmented reality (AR). To evaluate our system, we developed three game applications-a first-person-shooter game ...
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Xiruo WangAuthors Info & Claims
