5.1. Experiment 1: AR smart glasses and MR for trench data collection

Our first experiment was also our simplest: We wanted to see if AR smart glasses could serve as the primary platform for basic hands-free data entry during excavation, incorporating database entry, 2D photography, and continuous narrative journaling. The goal is to replace the smartphones we use for collecting these types of data, our main data, at the trench. We primarily focused on testing the Vuzix Blade 2 device during this experiment, but we also ran some basic tests for journaling with the other two MR devices. We began by trying to use our existing custom smartphone database app and the Evernote app on the Blade 2, but we have since rebuilt our custom app for the Blade 2’s interface. Through these initial experiments, we have learned about the possibilities of the hardware, as well as about the challenges that we still need to overcome during future development and testing.

We currently depend on Android smartphones for fundamental data collection at the trench, using a custom app to record basic information about our distinct excavation contexts, to photograph the contexts and find bags, and to keep a running freeform daily journal in Evernote. We additionally collect 3D photogrammetric data with dedicated cameras and differential global navigation satellite systems (dGNSS) equipment, given the deficiencies of smartphones, but that type of data was not a focus of this experiment. We have run into some challenges with using smartphones during excavation. For example, excavators must pause digging to remove the phone from their pocket to work in the database or add to the journal, so work is interrupted and journal writing happens less often than it should. For convenience, the phone is sometimes left out on the ground, where it may be covered with dust, potentially blocking or even scratching the camera, and there is always a chance someone could step on the phone. Smart glasses could both enable hands-free data entry and review, while at the same time providing some protection for the eyes from dust and the sun (Vainstein, Kuflik, & Lanir 2016). Thus, we took the Vuzix Blade 2 to the trench over the course of several days, trying the device when the sun was low in the early mornings as well as during the bright mid-day sunlight (Figure 2; Figure 3). The Blade 2 connected to the internet using mobile hotspots from our smartphones at the local 4G speeds.

First, we attempted to install our existing custom smartphone app, which allows database entry and photography, onto the Blade 2. This device runs the older Android version 11, compared to our phones running version 12 or 13. Although the app loaded, the small screen has many limitations, and performance and interaction were especially challenging (Figure 4). The app expects touch screen interaction, but the Blade 2 can only either scroll among input boxes or use a mouse-cursor, and typing characters is cumbersome. The Blade 2’s build-in functionality for a mouse-cursor was laggy and crashed. Although we found that the original smartphone version of our custom app was practically unusable, it is a relatively simple client that mostly relies on remote web service calls to our central server. Thus, after the summer, our programmer has been iteratively updating and testing a new version of this custom database app designed specifically for the Blade 2. The main change has been the simplification of the user interface, which now first presents only a selection of the trenches. The user scrolls through these options using the right finger, finally clicking to select a trench. The next screen then provides the ability to either create a new context in that trench or choose from existing contexts, with the latest ones displayed initially on the screen, and the rest accessible by scrolling down. Once a context is selected, the next screen allows the user to update database information such as date opened or closed, context type, or a short text description. We have had some success training a Large Language Model (LLM) to support voice-recognition for adding and updating these textual descriptions. Finally, a user can take 2D photographs of the context or its find bags in the field, and these are uploaded to our server. We are in the testing phase for this new app, continuously improving the user experience and stability, with plans to try this in the field during a future season.

We also tried to install the Evernote app on the Blade 2, but again found it barely functional for two basic types of data entry (Figure 5). First, we purchased a license for Vuzix’s speech-to-text (STT) program, and we were able to speak some small narrative as text entry, though it was not accurate enough for regular use. As mentioned before, we have since developed greatly-improved LLM-supported voice-recognition, but we will need to find a way to integrate this with Evernote. Second, since the Blade 2 can capture forward-facing video and still images with its 8-megapixel camera, we were able to load a photograph into Evernote. The built-in voice activation functionality also worked well for basic navigation and operating the camera, by simply saying ‘Vuzix, take a picture.’ Although it does occupy the right hand, scrolling with the right finger did function well and is still quicker and easier than taking the phone out of a pocket, but we need further experimentation when wearing work-gloves. We are considering to create our own custom journaling software in the future to replace Evernote to better fit this platform.

In terms of being able to use the Vuzix Blade 2 on a regular basis, our experience indicates that this form factor would work very well for use throughout the excavation day at the trench. The device is comfortable, has good internet connectivity, a decent camera, and interaction via voice or finger is sufficient. Perhaps the main downside at the moment is the screen, but our latest software is specifically designed for its limited size and interactivity. We may also find that looking through the screen with only the right eye could become uncomfortable after long periods of use, though most of the time the screen could remain off and, although we have not fully tested battery life, it appears to last all day. Sunlight was a challenge for the Blade 2, but less than for the HoloLens 2, so the Blade 2’s screen could still be brighter. It helps to look at a dark background to see the screen. We are also working on two directions to improve this situation. First, we purchased add-on sunglasses-type shades that are placed on the exterior of the lenses. Initial testing indicates that this decreases the contrast between the real-world and the screen by decreasing overall brightness. Second, we are also planning to experiment with other types of smart glasses that have an actual non-transparent screen placed in the peripheral vision of the user, including by building our own smart glasses from parts.

Separately, we envision great potential for the use of such AR smart glasses for field-walkers during archaeological surface survey. Although less crucial to have hands-free data entry, there are times in our region when we find ourselves scrambling over difficult terrain with our hands. The Blade 2’s built-in camera, if paired with a global navigation satellite system (GNSS) position from a connected device, could support a smooth data collection workflow. Furthermore, the heads-up display could show a map with the walker’s real-time position, including marking the track already walked and helping keep team members within their designated sampling lines. In addition, with local wireless connectivity, the map could show the positions of other walkers to improve team coordination, and could even perhaps show an initial distribution of find locations made by the team.

Since our two MR devices have more capabilities than the Blade 2, we briefly experimented with using them for the same type of basic data collection within the trench. Although it would be less practical to wear the HoloLens 2 while excavating all day, we were able to successfully replicate some of our smartphone functionality. Specifically, we found third-party note-taking applications for the HoloLens 2 that used voice recognition and image capture (Figure 6; Figure 7). These functioned moderately well, but the voice recognition was not accurate enough for regular use. We also built our own custom software workflow to implement basic note-taking functionality within the Quest Pro (Figure 8). We experimented with Google’s online speech-to-text service, but this requires uploading an audio file and thus does not fit with our real-time writing needs.

5.2. Experiment 2: MR for reviewing 3D context models in situ

MR has the potential to fill a very important gap in archaeological practice, namely, the ability to see previously excavated remains placed precisely back into their original locations (Dilena & Soressi 2020). This could be for artifacts, both large and small, for architecture and other features, or for any volumetric excavation context. These would aid with understanding stratigraphy, comparing scale, planning how to excavate next, and generally thinking through a wide variety of important archaeological questions (Figure 9). The use of MR at the trench could also open possibilities for group collaboration. Multiple users could view the same 3D models in the same place at the same time, from different angles while still seeing each other, thus encouraging discussions (Sereno et al. 2022). Through today’s high-speed internet connectivity, it could even be possible for people not physically present to take part in the field discussions. If excavation projects would start sharing their primary 3D data openly, we could imagine directly comparing walls, features, and spaces among different sites at full 1:1 scale (Cobb and Nieminen 2023:244). Users could also use MR to view geophysics data on the surface to determine where to excavate next.

Thus, for this second experiment, we tested the viewing of 3D models of previously excavated contexts replaced back in situ in the trench. For this experiment, we used the two MR devices that enable such spatially-aware 3D immersion. At our excavation, we capture 3D models using photogrammetry whenever we complete the excavation of a distinct volumetric context (Roosevelt et al. 2015). Our programmer developed custom software with a small selection of our actual 3D models to display in the trench using the HoloLens 2 and the Quest Pro (Figure 10). The user interacts with these apps through intuitive hand gestures to scroll through virtual menus that appear to float before them in the air. An important step in this process was geolocating the models in their proper positions, even though the headsets do not know their location on Earth, so we developed an anchoring solution. We spent several days at the trench investigating what was possible, how things looked, and documenting the current limitations. These initial experiments have enabled us to plan future work that also integrates more closely with our database.

Within the top trench at the Vedi Fortress, the entire 10-meter western edge originally contained an Early Medieval (Late Antique) rebuilding of the site’s main upper fortification wall. This wall was about 2.75 meters thick, made of medium sized stones, with only several courses preserved. This structure was fully uncovered by the end of the 2022 season but removed in 2023 for deeper excavation. Our experiments with MR therefore replaced this crucial feature back into the trench to compare it to the walls of the earlier, deeper layers (Figure 11; Figure 12; Figure 13; Figure 14). We then placed the 3D model of a smaller adjoining wall, also from the Early Medieval period, that helped form a storage cellar built against the inner face of the fortification (Figure 15; Figure 16). Finally, we situated the 3D models of the inner and outer surfaces of a large ceramic storage vessel (pithos) that may have been in this storage cellar (Figure 17). We imported all these 3D models using either the .obj or .glTF (Graphics Library Transmission Format) file formats. This basic experiment worked well for both devices, but each had its strengths and weaknesses described in the following paragraphs, including sunlight visibility, field of view, spatial awareness, and limited boundary extent.

The HoloLens 2 screen is not bright enough to be visible under full sunlight. The only way we could use this device on the site was at dawn or dusk. Therefore, a small team periodically went out to the site at 5 am as the sun began to rise, providing about an hour of usable light levels (Figure 18). As mentioned above with the Blade 2, we are also planning to add sunglasses-type shades to the HoloLens 2, which have to be custom produced. The Quest Pro, on the other hand, has no problem with sunlight since it simply passes through the view from the cameras, so we could experiment all day. Although you can tell you are looking through cameras, everything is clear without much distortion and the graphics are sharp and can even be fully opaque (Figure 17).

Another major issue with the HoloLens 2 is the restricted field of view. This means that when you are in the trench, you must be looking almost directly at the digital 3D models in order to see them displayed on the screens. No digital models will appear in your peripheral vision, and the transition is an extremely abrupt rectangle. This makes the experience much less satisfying because you still see the entire real trench, but your graphics are cut off around the edges. The Quest Pro restricts your entire field of view, both of the real and of the virtual worlds, which provides a more seamless experience, even if constricting your natural peripheral view (Figure 19).

At first, we manually scaled and oriented each 3D model while wearing the devices in the trench, and then we were able to view the models from different angles as they remained stationary within the real world (Figure 20). We could use hand gestures to ‘grab’ two edges of each model to rotate them or scale them by pulling our hands apart. However, we geolocate our photogrammetry models using targets measured with differential GNSS (Cobb, Earley-Spadoni, & Dames 2019), which means each 3D model can be automatically placed within the trench. Although the HoloLens 2 has spatial awareness, it does not know its position on Earth or even which direction it is facing. Thus, the only spatial information we can directly use from an original 3D model is its scale, with one digital unit becoming one meter in real life upon importation. For the Quest Pro, we need to set all the parameters of the model, including the scale, position, and orientation, when placing it into the real world. Although positioning each model manually is an option, this can lead to inaccuracies. Thus, we developed a solution that allows us to first establish the global coordinate system within the trench. We created three 3D models of flags to represent the locations of three corners of the trench and labeled each corner with its Universal Transverse Mercator (UTM) coordinates. When we brought the devices out to the trench, we placed these three virtual flags on the actual three corners of the trench (Figure 21). For the HoloLens 2, we actually only need to place two corners to set the horizontal position and orientation. After doing this, the local coordinate system of each device was aligned to the actual real-world UTM grid, and thus each of our 3D context models could be immediately imported with its proper location, orientation, scale in the trench. Hopefully, future MR devices will rely on internal and external dGNSS so that we can automatically and immediately load models into their real-world locations.

The placement of models in the Quest Pro was complicated slightly by the limited extent of the Guardian boundary. Although online forums claim that the boundary can grow up to 15 × 15 meters, and can even be disabled during pass-through, we were unable to achieve either of these situations. Instead, the largest practical Guardian extent we could lay out was only 6 × 6 meters, well below the full 10 × 10-meter extent of our trench. Since the Guardian acts as the spatial anchor for all models, any attempt to shift the extent to view other parts of the trench would cause all the models to move in unison (Figure 22). The HoloLens 2 does not have this limitation so we could see the full 3D models in situ whenever we looked directly at them.

For our next step, we plan to develop software that will integrate with our existing data management system, allowing users to load any 3D model while at any trench. To prepare 3D models, we synch raw image files for photogrammetry to our cloud-based Windows Remote Desktop server using OneDrive each afternoon. By the next day, trench supervisors can employ Agisoft Metashape on the server to initiate the building of each context’s 3D model, about 2–3 per trench per day. Since a typical model, built at high quality, may contain about a million polygon faces, we down-sample each model to around one hundred thousand polygons or less, while maintaining a high-resolution texture, to balance visual quality at a useful scale with efficient file size. Given the limitations of the screen resolutions in the current headsets, these down-sampled models already provide a satisfactory viewing experience. Finally, we export the models as .obj files, making them available on the server by about two days after excavation, though we may later focus on developing ways to speed up this process if necessary. To get the models to the headsets, we are working on expanding our existing web service, which allows remote clients to access our central database, to include the serving of 3D models. Since each .obj file is usually up to 100 MB in size, this may raise concerns about the limited 64 GB local storage capacity of the HoloLens 2. However, it should be noted that a user would probably only be able to see about a dozen or so 3D context models at one time, with others hidden behind the visible models or in other trenches, so this should not be an issue even though most trenches have over 200 total context models. We are designing an interface that will allow users to load contexts on demand, which should take less than a minute even for the largest models to be downloaded over a 4G mobile connection. In addition to the inherent geolocational information in each 3D model, other metadata about each context can also be quickly retrieved from the central database using existing web service calls. The user interface we are now developing for the MR headsets will enable users to select which models to view through intuitive spatial interaction, based on their own location in the trench. This will open new and interesting human-computer interaction research that will be the subject of future analysis about our field experiments.

5.3. Experiment 3: The potential for MR-guided excavation

Our third effort was much more experimental, as we attempted to push the limits of what is possible and think of new ways of doing archaeology. We see opportunities for MR devices beyond just displaying data, where they could have a central role as active tools directly in enhancing excavation work. In our excavation, we often need to cut clean, flat surfaces to see better. This may be for the baulk sections but is often also for tracing horizontal layers, either at the end of removing basic dirt fill contexts or to make features or in situ finds more visible. Therefore, we had the novel idea that we might be able to use the MR headset to guide us in cutting perfectly flat surfaces at any angle. The same idea could be used, even more fundamentally, to carefully control the vertical ending depth while digging selected contexts. For example, if we wanted to excavate a horizontal layer exactly 20 centimeters deep, we could use the MR device for such depth guidance. Only the HoloLens 2 is currently capable of this functionality since it creates a polygonal model by 3D scanning existing real surfaces in real time. Thus, our programmer built a depth-guidance system to allow us to place a simple red flat plane horizontally (floor) or vertically (wall) within the real-world trench (Figure 23). The user determines where they want to excavate a flat surface and then they move the plane into place, even pushing it below the ground.

We experimented with using this depth-guidance functionality first to visualize the flatness of the already excavated south baulk of our trench (Figure 24). In the image, the red plane is visible where the real-world surface is located behind it and invisible where the real-world geometry covers that part of the plane. Thus, you can see that certain portions of the baulk section could be leveled better, though most of the complex topography is caused by stones. After examining the baulk, we next moved to using the MR depth-guidance system during actual excavation. We selected a raised area of dirt that we wanted to cut down to a surrounding flat surface about 15 centimeters lower. One team member first set the flat plane to the desired depth and then he actually dug while wearing the HoloLens 2, watching for the red plane to appear in a pit he dug from the center of the raised area (Figure 25; Figure 26). This experiment was a success as he ended when he saw the red plane, thus producing a precise depth of excavation. This depth-guidance function already works surprisingly well, even within the limitations of the existing technology. One could imagine other such uses for MR to guide an excavator through forming geometry within the real world or predicting what will come next in the trench. For example, perhaps the MR could indicate the likely line of an existing wall, virtually connecting the excavated portion to the unexcavated portion for the excavator.

Such experiments that push the limits are necessary if we are to fully test the potential of the hardware. Another possible use for MR devices would be in the training of archaeologists, with virtual guides and even artificial intelligence (AI) enabled learning (Deru et al. 2023). This might be similar to the way Brondi et al. (2016) attempted to teach fine motor skills with the hands for heritage craftwork. Furthermore, similar to the experiment by Gaugne et al. (2019) to help excavators see ‘inside’ an urn, we believe that MR can directly find applications for guiding digging on-site, including tracing of geophysics data.

5.4. Experiment 4: MR for artifact analysis

The ever-increasing quantity of available 3D data about archaeological artifacts opens new opportunities for using MR during object analysis. In our final experiment, we set out to specifically try to compare real pieces of pottery with digital 3D models of related sherds. Using hand gestures that are tracked by the HoloLens 2, we were able to hold, through pinching, a virtual pottery sherd. We could then rotate and move the sherd within the real world. Thus, a team member experimented with puzzling together two large ceramic base sherds using the HoloLens 2 – one real and one virtual (Figure 27; Figure 28; Figure 29). This works reasonably well. However, the transparency of the virtual sherd makes the edges somewhat difficult to see as they can sometimes be obscured by the real sherd. We also tried joining sherds based on surface decoration (Figure 30) and comparing the profile of a real rim sherd with the profile of its virtual version (Figure 31). The HoloLens 2 actually worked better for this indoor experiment compared to those that took place in the trench, given the controlled lighting. In addition, the limited field of view was less of an issue since the sherds were close to the user and relatively small.

We could imagine using this method with other material classes like comparing lithic shapes or rejoining sculptural fragments. We should be able to compare or join the real objects from our site to virtual 3D objects scanned from other sites, storage areas, and museums, including those already shared online (Kobeisse 2023). Often times, such objects are difficult to access, either because of travel or permission restrictions. Another idea is that a real pottery sherd could be held within a virtual whole vessel to determine how well the shape matches or to determine the possible position of the sherd within the larger vessel. All these methods enable the type of embodied interaction with objects that specialists are used to, but they open new possibilities to work with precise data about objects that are remote or otherwise difficult to retrieve.